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Designing Education Projects Manual and Course 


The National Oceanic and Atmospheric Administration (NOAA) has a proud tradition of 
educating the public about the environment. That tradition carries on today with an agency-wide 
emphasis on increasing the public’s environmental literacy. NOAA’s education coordinators 
play a critical role in this effort. Their education work supports the four mission goals identified 
in the NOAA Strategic Plan. Ultimately, the effectiveness of these education projects is essential 
to NOAA’s overall success. 

This manual and the course that it supports were designed to assist education coordinators as 
they design and implement their projects. By using the strategies and tools it describes, 
education coordinators can understand how their projects are working and if they are meeting 
expectations. They will be able to clearly articulate project objectives, outputs, and outcomes; 
quantify changes in the knowledge, skills, and behavior of their target audiences; and develop the 
mechanisms needed to track tangible progress toward reaching NOAA’s education goals. 

I hope you find this manual and the Designing Education Projects Course helpful and you are 
able to use them to design more effective education projects supporting the agency’s mission 
goals, and increase environmental literacy among our Nation’s citizens. 



Introduction 


A considerable amount of time, effort, and other resources go into the development 
and implementation of education projects. Quite obviously, the goal is to create 
effective projects that can serve as models of excellence. Whether the project is 
an hour-long endangered species talk, a family festival, a severe weather awareness 
workshop, marine resources monitoring, or a community forum, the aim of providing 
quality educational experiences remains the same. 


Who Should Use This Manual? 

This manual has been developed to help project managers and education coordinators 
who design education projects take the development process to a new level of excellence. 
The manual walks through the basics of needs assessment, project planning and 
implementation, and evaluation. This information is intended to answer questions about 
the project development process and to be used as an overall project improvement tool. 

A systematic approach to the overall project planning and implementation process 
is outlined. Education coordinators are encouraged to take a step back and make 
considered decisions that will result in increased effectiveness. Specifically, using the 
process described in this manual will help you: 

• Match agency’s needs and capability: Throughout the process, you will be asked to 
consider how the project addresses the agency's mission. Knowing how the project 
"fits" within the agency's priorities will ensure strategic use of resources and help you 
build internal support for the project. You will be able to articulate why this project is 
important and why NOAA should support it. 

• Set realistic and meaningful goals, objectives, and outcomes: From the development 
of the needs assessment to designing the final evaluation and everything in 
between, you will know what you want to accomplish and if you actually 
accomplished it. You will set objectives that are measurable and worth measuring. 
You will focus your project on outcomes that make a difference and ensure that a 
direct relationship exists between outcomes and project components. In a time of 
increasing attention to accountability, you will be able to document educational 
impacts. 

• Use limited resources wisely: By identifying measurable objectives based on well- 
thought-out and researched priorities, projects will be focused and resources will be 
used efficiently. At various points in the project design process, you will be asked to 
inventory existing materials and programs. By taking stock, you will avoid re-inventing 
the wheel. Judiciously adapting or adopting materials saves both time and money. 

• Design effective and sustainable projects: Projects that are designed with best practices 
in mind are more effective: continuous project improvement becomes integral to 
the process, evaluation is not left to the end, stakeholders' needs are consciously 
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addressed throughout, and credibility is built. When decision-makers and others see 
results and meaningful partnerships established, projects are truly sustainable. 

• Enhance the learning process: In the end, education projects are developed because 
of the learner. Projects are developed because we want participants to gain specific 
knowledge and skills. Education projects are developed to promote public safety and 
the development of environmental and scientific literacy. Careful attention to the design 
and implementation of an education project will be reflected in learner outcomes. 

It should be noted at the outset that this manual outlines an ideal process for the design 
of high quality education projects. Developing appropriate budgets and schedules 
are, obviously, key to the ultimate success of the education project. Without proper 
attention to budget details, a project may never make it beyond the early stages of 
planning. Similarly, poor scheduling may mean that materials are not ready when 
needed or evaluation opportunities are missed. Although both budgeting and scheduling 
will impact the quality of the project, the focus of this manual is on the design of the 
education intervention and its evaluation. 

Each project is different, however, varying in scope and duration. Materials developed 
for a one-shot event at an elementary school will be very different from those developed 
for a community group that meets on a regular basis. Projects vary in the amount of 
resources (human as well as monetary) available. It is unrealistic to expect all education 
coordinators to follow all of the recommended steps. The steps outlined here are meant 
as a guide. Use them as a template to inform your decision-making. Use them to help 
ensure the development of effective education projects. 


A Word about Programs and Projects 

In everyday speech it would not be unusual to hear someone talk about the NOAA 
Education Program or for a presentation given to a group of fifth grade students at a 
local school to be described as a program. Similarly, it is not unusual to hear project and 
program used interchangeably. For purposes of clarity and consistency, "program" and 
"project" are used in this manual in very specific, distinct ways. 

Programs derive directly from the agency's mission and represent a coordinated and 
systematic effort to address that mission. Programs support NOAA's Strategic Plan and 
goals. A set of projects, taken together, reinforce a program. In turn, a series of activities 
are devised to address project goals and objectives. Projects are focused on specific issues 
and audiences. The following graphic might help to illustrate the distinction. 
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Project Development as a Cycle 

Project development requires a commitment to a 
systematic, iterative process of assessment, design, 
implementation, and evaluation. Whether the process 
is described using a flow chart or through text, it 
is important to remember that the process is not 
linear. Although each stage or phase of the project 
development and implementation process can be 
described (and will be in this manual), the stages 
are not discrete. These steps overlap and interrelate; 
they provide a dynamic and flexible guideline for 
developing effective projects efficiently. Project 
development is a cyclical process in which the results 
of one phase become the starting products for the 
next phase. 


Programs vs. Projects 



THE PROJECT DEVELOPMENT CYCLE 


I " 4 


Needs 

Assessment 


Evaluate 





Design 


and the project's 
approval at each 


At each step in the process, the 
project team reflects on decisions 
and activities that have gone on 
before, and assesses successes and 
the need for course corrections. 
Underlying project planning and 
implementation is the premise that 
learning about how the project works 
takes place continuously throughout 
the process and that learning is fed 
back into the system. 

It should be remembered that not all 
projects are alike. How the project 
development cycle plays out will 
depend on the politics surrounding 
the project, funding requirements, 

degree of complexity. For example, some projects will need review and 
stage, while others may only require limited oversight within the agency. 


Implement 




How is this Manual Organized? 

For the convenience of the reader, the major sections of this manual will follow the 
project development cycle. It is divided into four major parts: 

Part I Needs Assessment 

Part II, Project Planning and Implementation 

Part III. Project Evaluation 

Part IV. Data Collection Instruments 
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Parts I through III utilize a project development model created by Bennett and Rockwell 
(1995, 2004). Targeting Outcomes of Programs or the TOP model integrates evaluation 
within the project development process. When planning an education or outreach project, 
education coordinators start at the upper left-hand side of the model by describing the 
social, economic, and environmental (SEE) conditions that need to be addressed. As the 
education coordinators work their way down the programming staircase they identify the 
program design elements necessary to achieve the desired SEE conditions. At each level 
of project development, outcomes are identified and later used as benchmarks to help 
evaluate the extent to which they were achieved. Project implementation and outcome 
evaluation ascend TOP's programming staircase on the right-hand side of the model. Part 
IV describes a variety of common data collection instruments or tools followed by a series 
of appendices that provide a more extensive consideration of specific topics. A glossary of 
technical terms, especially those related to needs assessment and evaluation, and a list of 
references are included at the end after the appendices. 


TOP Model’s Programming Staircase 


Project Development Project Evaluation 



Source: Bennett, C. and Rockwell, K. (2004). Targeting Outcomes of Programs (TOP): A hierarchy for targeting outcomes and evaluating 
their achievement. Retrieved January 2009, from University of Nebraska-Lincoln website: http://citnews.unl.edu/TOP/index.html 


V. 
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Throughout this manual, examples and case studies are provided. These examples are 
based on existing education projects run by or supported by NOAA. They offer a glimpse 
of how other education coordinators handled a particular step in the design process. 
Hopefully these examples, by being rooted in real world experiences, will help you 
see how the project development cycle can best be fitted to your work. As you read the 
examples and their accompanying text, please remember that this manual outlines a 
comprehensive project development process. Depending on the scope and scale of the 
education project you are proposing, all of the steps may not be followed completely. 

The following education and outreach projects are used to demonstrate different project 
planning, and evaluation concepts and approaches. Each project is in a different stage 
of development and has contributed either directly or indirectly to the content and 
examples provided throughout the manual. While not all of the examples followed the 
TOP Model, they provide insight into how NOAA education coordinators developed 
their projects. 

• Alaska Fisheries Science Center (AFSC) Committee on Outreach, Education and 
Diversity (COED) Professional Development Project. National Marine Fisheries 
Service. Seattle, WA 

• Aquatic Invasive Species Exhibit Development. Oregon Sea Grant College Program. 
Corvallis, OR 

• Bay Watershed Education and Training (B-WET). NOAA Chesapeake Bay Office. 
Annapolis, MD 

• Costal Training Program. Padilla Bay National Estuarine Research Reserve. Mt. Vernon, 
WA 

• Florida Hazardous Weather Awareness Week. Florida Division of Emergency 
Management and National Weather Service. Melbourne, FL 

• MERITO (Multicultural Education for Resource Issues Threatening Oceans). Monterey 
Bay National Marine Sanctuary. Monterey, CA 

• Nab the Aquatic Invader! Be a Sea Grant Super Sleuth. Illinois/Indiana Sea Grant 
College Program. Urbana, IL 

• Needs Assessment for Ocean Education in North Carolina. North Carolina Sea Grant 
College Program. Raleigh, NC 

• Office Leadership Development Course. National Weather Service. Springfield, MO 

• Science Spectrum Weather Center. National Weather Service. Lubbock, TX 

• Southeast Alaska Cloudburst Chronicle. National Weather Service. Juneau, AK 

• Turn Around Don't Drown!™. National Weather Service. San Angelo, TX 

• Volunteer Weather Spotter, SKYWARN Training and Communication. National 
Weather Service. Lubbock, TX and Portland, OR 
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Introduction 

The design of an education project often begins with a phone call or a casual 
conversation. Some one suggests that a particular audience should be reached (e.g., 
school children, community leaders, families), a particular topic remains misunderstood, 
or a particular environmental issue needs to be addressed. The conversation soon ends 
with the suggestion that education may well be part of the answer, and that perhaps a 
project should be developed that would reach the target audience or cover the identified 
topic. But once the seed of an idea has been proposed, what should be done next? By 
conducting a needs assessment, education coordinators can focus their energies efficiently 
and develop an effective project. 


What is a Needs Assessment? 

On the surface, defining a needs assessment is quite simple - it is the effort to assess 
the need for a project or other activity. An education needs assessment establishes 
the need for a particular project by systematically examining audience interest and 
knowledge, agency mission, authorities and capability, and the significance of particular 
environmental conditions or issues. 

It may be helpful to visualize needs assessment as a form of gap analysis. In this case a 
needs assessment can be seen as a systematic exploration of the divergence or discrepancy 
between the current situation or level of services ("what is") and the desired situation 
or level of services ("what should be"). In analyzing the "gap," project team members 
can begin to identify problems, opportunities, strengths, challenges, and possible new 
directions. Through a careful analysis of the existing "market," and an assessment 
of organizational strengths, a more strategic understanding of gaps and potential 
opportunities emerges, reducing the chance of duplicating the efforts of existing programs 
or becoming sidetracked. Through this process, it is also possible to identify and develop 
strategic partnerships to better reach underserved audiences. As Kaufman and English 
(1979, p. 31) suggest, a needs assessment is " ... a tool for determining valid and useful 
problems which are philosophically as well as practically sound. It keeps us from running 
down more blind educational alleys, from using time, dollars and people in attempted 
solutions which do not work." 

The results of this analysis provide a context to determine an organizational niche 
and identify target audiences. For NOAA, a needs assessment should also include a 
process of self (agency) assessment of internal strengths and weaknesses relative to the 
agency's mission and the environmental issues it seeks to address. Importantly, a needs 
assessment provides education coordinators with well-documented and considered 
evidence to support the project design process. A needs assessment does not merely 
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justify a proposed project after critical decisions have been made, but helps to establish 

why a project may be needed, and provides critical data and information about the 

ultimate structure of a project. 

As might be expected, needs assessment takes place prior to undertaking the project. 

Needs assessment: 

• Identifies and determines the scope of the social, economic, and environmental 
conditions or issues that need improving. 

• Gathers information/data about the gap between the current and desired level of 
audience knowledge, attitudes, skills, aspirations (KASA), and behaviors. 

• Helps confirm or negate assumptions of audience characteristics and appropriate 
content; defines goals and objectives; ensures goals and objectives are aligned with 
the agency's strategic plan and other planning documents; and identifies stakeholders 1 
and potential collaborators. 

Questions that might be addressed by a needs assessment include: 

• What are the nature and scope of the problem? Where is the problem located, whom 
does it affect, and how does it affect them? 

• What is it about the problem or its effects that justifies new, expanded, or modified 
projects or programs? 

• What feasible actions are likely to significantly ameliorate the problem? 

• Who is the appropriate target audience(s)? 

• Who are the stakeholders and sponsors? 


Why is Needs Assessment Important to Project Design and Implementation? 

The initial idea spurring the development of an education project may have come from 
a variety of places. Casual conversations among co-workers, serendipitous opportunities 
such as the availability of grant funding, the publication of a scientific report or the 
long-term desire to enhance strategic partnerships may all spark the project development 
process. For example, sharing a facility with the Spectrum Science Museum in Lubbock, 
Texas presented the Weather Forecast Office (WFO) with an opportunity to showcase 
weather sciences. The WFO took advantage of the their shared space and worked with 
museum staff to design an interactive, weather-related exhibit, the Weather Center. 

In the enthusiasm and excitement of the moment, however, it must be remembered that 
not all projects are timely or appropriate. Conducting a needs assessment allows project 
managers to take a step back and systematically consider whether or not there is a gap 

1 The term "stakeholder" means anyone with an interest in the goals of your project. Stakeholders may be the participants 
in your education project or customers who use your product. They may be project sponsors, those individuals or agencies 
to whom you are accountable through your project's outcomes. They also may be partners with similar goals who may or 
may not receive a direct benefit from your project. 
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in existing services or materials, and if so, the nature of that gap. There are additional 

benefits of including needs assessment as an integral component of the project design 

and implementation process: 

• Serving the Audience: Ultimately, the people who participate in education projects 
benefit from needs assessments. Education services are more targeted and delivery 
systems are designed better to reach their intended audiences when founded on 
data, rather than hunches. Because needs assessments systematically gather data, 
previously unexpressed needs can be uncovered and, consequently, the audience can 
be better served. 

• Setting Priorities: For any one problem or issue, the "need" is rarely one-dimensional. 
A needs assessment helps project planners to systematically describe the audience(s) 
impacted by the issue and their relationships to the issues as well as the underlying 
causes. With this level of information, administrators and project planners make 
informed decisions about which possible solution or combination of solutions can 
best address the need. Faced with a long wish list, the needs assessment provides the 
data to develop criteria necessary for priority setting. 

• Re-inventing the Wheel Any time a new project is initiated there is some danger that 
it duplicates efforts already taking place elsewhere within the agency or the wider 
community. A needs assessment will determine if materials or projects developed 
elsewhere can be adapted or adopted to the new situation. Considerable time, effort, 
and resources can be saved by taking stock of what already exists and not falling to the 
temptation of creating something "new" for its own sake. 

• Resource Allocation No matter what the problem or issue, project planners must 
confront the budget process sooner or later. Administrators will want documentation 
to substantiate decisions about which proposed projects should be fully funded, 
postponed, or rejected. By documenting the need for a project and providing data- 
based evidence of how the project will address the need, project managers can make a 
reasoned case and assist administrators in allocating resources appropriately. 

• Coalition Building: Well-designed needs assessments are highly participatory. Not only 
are agency staff members involved in setting project priorities, but a wide variety of 
stakeholders are identified and involved at each step in the process. Participation can 
demystify a project and help ensure greater buy-in from agency personnel, partners, 
and potential audiences. If true coalitions are to be built, stakeholder participation 
cannot be approached as a public relations ploy. The needs assessment must be open 
and welcoming of ideas and not designed to validate a pre-determined course of 
actions. 

• Strategic Planning: In defining the gap between what is and what is desired, a needs 
assessment can be a powerful strategic planning tool. First and foremost, a needs 
assessment focuses project planners' attention on the end goal. By mapping out the 
current situation systematically, planners have the data to make decisions about 
realistic and meaningful goals. Additionally, a project needs assessment can serve as 
an important supporting document to the Program Baseline Assessment in NOAA's 
Planning, Programming, Budgeting and Execution System (PPBES). 
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Spotlight on MERITO 

Multicultural Education for Resource Issues Threatening Oceans 


MERITO’s mission is to help advance ocean literacy among culturally diverse residents and pro¬ 
mote a culturally inclusive ocean stewardship ethic by engaging diverse communities to protect 
coastal and marine ecosystems. 

In response to the growing social, economic, and environmental (SEE) issues threatening the 
oceans off the coast of central California, the Monterey Bay National Marine Sanctuary (MBNMS) 
undertook a needs assessment to determine how they could more effectively conduct community- 
based education and outreach. Assessing the current situation was the first step in the process. A 
literature review of the demographics of California’s central coast called attention to the culturally 
diverse Hispanic communities that surround the Sanctuary. According to the US Census Bureau, 
over 47% of Monterey County’s population is Hispanic. In addition, this region of central California 
is considered a nationally significant agricultural center, growing more produce than any other 
region in the United States. Studies show that Hispanics in this area earn significantly less and 
achieve lower educational levels than other ethnic groups making this population less likely to 
acquire the desired understanding of environmental issues threatening local marine resources. 

For MBNMS, the desired situation was to increase public understanding of specific ocean-related 
issues within sanctuaries and promote community participation to address these issues. 

It was evident that the Sanctuary needed to develop projects that targeted their Hispanic constitu¬ 
ents. MBNMS staff conducted interviews with thirty community leaders representing local school 
districts, universities, non-profit organizations, government agencies, and the farming community. 

These stakeholders provided valuable information resulting in a list of critical needs and guide¬ 
lines for a multicultural education program. With attention to the agency mission and capabilities, 
the needs were prioritized and grouped into themes, which guided the MERITO program design 
and strategic plan. The resulting program focuses on four core components: 1) Community-based 
ocean outreach; 2) Site-based ocean outreach; 3) Professional development and internship pro¬ 
gram; and 4) Bilingual outreach products and media. Within each component, a series of projects 
are conducted which address the goals of the MERITO program. 

Source: Monterey Bay National Marine Sanctuary (2001). Multicultural Education Plan: Hispanic Component “M.E.R.l.T.O." Mul¬ 
ticultural Education for Resource Issues Threatening Oceans. Monterey, CA: NOAA, Dept, of Commerce. For more information on 
MERITO visit the Monterey Bay National Marine Sanctuary website at http://montereybay.noaa.gov/educate/merito/welcome.html 

V____ J 

Planning a Needs Assessment 

As mentioned earlier, a needs assessment can be seen as a systematic exploration of the 
divergence or discrepancy between the current situation or level of services ("what is") 
and the desired situation or level of services ("what should be"). In analyzing this gap, 
project team members begin to identify problems, opportunities, strengths, challenges, 
and priorities. The needs assessment process can be time consuming and, as with most 
processes that involve multiple stakeholders and multifaceted issues, it can be complex. 
The following is an outline of 13 steps involved in conducting a needs assessment. The 
outline is intended to break down a complex process into manageable steps. Please 
recognize, however, that in providing an overview of the process, nuances and detail 
are necessarily omitted. Similarly, recognize that determining how much of the process 
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should be followed depends on the project. The time and effort involved in conducting 
a needs assessment must be balanced against the time and resources available to be 
invested in the project as a whole. A project that involves developing a 30-minute 
weather awareness activity for local kindergarten classes suggests a very different needs 
assessment process than a project that involves developing a comprehensive weather 
awareness curriculum for pre K-12 students throughout the state. The steps outlined 
below should be adapted to meet the specific needs and resources of the project. Look at 
the examples provided throughout this section as an indication of how others have used 
needs assessments in situations similar to yours. 




Where do ideas for education programs come from? 


Although it would be great if all programs grew out of a true needs assessment, most education and 
outreach programs start with someone having an idea. In these cases, the needs assessments look 
at gaps the idea could fill or ways the idea could be adapted to fit a particular audience. So who 
comes up with these ideas? 

Most ideas for educational programs come from the people most closely involved in a particular area 
- it’s “my passion,” or “my field,” or “my experience” that determines the topic. Sometimes the idea 
for a program comes down from ‘above’ and is a passion or pet project of someone higher up in the 
organization. A third source of ideas is from a friend, a stakeholder, or a member of the public at large. 

Often these ideas work very well. For example, one National Weather Service (NWS) Weather Coor¬ 
dination Meteorologist (WCM) based his project on what he knows best, disaster preparedness. This 
WCM recognized the need for improved education about flood and flash flood safety for people who 
are driving or walking into flood waters. His experience led to Turn Around Don’t Drown!™, a suc¬ 
cessful education campaign that continues to garner support and funding from many partners and 
stakeholders. In another situation, the Sea Grant project coordinators for Nab the Aquatic Invader! Be 
a Sea Grant Super Sleuth recognized the need for teachers and students in the Great Lakes region 
to get involved with invasive species issues. Because the topic is considered a National Priority Area 
of Sea Grant the project coordinators were able to combine their understanding of teacher needs in 
science and technology along with their backgrounds in education and communications to develop 
an award winning interactive, science-based education website. 

Both of these cases worked well because the topics of the programs were closely tied to the mis¬ 
sion of the agency. However, many programs fail to ensure a close and ongoing tie between the 
outcomes of the educational effort and the purposes of the agency. Sometimes the individual charged 
with developing or implementing an educational program thinks of direct delivery to school groups 
as the only approach. In many cases, the agency or division may not have an education mission, or 
its education mandate could be to reach the larger public and not a particular school grade. Such a 
mismatch can lead to “mission creep” and an inability to explain why a program is worthy of ongoing 
internal support, even if there is evidence that the program is good. When budgets are tight, mission- 
focused educational programs are even more important. An idea based on an individual’s passion can 
be great, but the outcomes of the program must also closely relate to the outcomes of the agency 
and division’s missions, or the program will be hard to defend. 

V___ J 
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“...some factors that are basic to a successful needs assessment: 

• Keep in mind the value and necessity of broad-based participation by stakeholders. 

• Choose an appropriate means of gathering information about critical issues and other data. 

• Recognize core values in the group whose needs are being assessed. 

• Needs assessment is a participatory process; it is not something that is “done to” people. 

• Needs assessment cannot ignore political factors. Some people may view the process as caus¬ 
ing a loss of control. The priorities derived may be counter to entrenched ideas in the system. 

• Data-gathering methods by themselves are not needs assessments. The needs assessment is a 
total decision-making process, in which the data are but one component.” (p. 17) 

Source: Witkin, B.R. and Altschuld, J.W. (1995). Planning and conducting needs assessments: A practical guide. Thousand Oaks, 
CA: Sage Publications. 


Needs Assessment in a Nutshell 


PLANNING 


Step 1. Set the focus, refine the issue(s) and identify the stakeholder(s) 

Step 2. Establish the planning team 

Step 3. Draft a plan for carrying out the needs assessment 

Step 4. Use the TOP model to direct data collection efforts 

Step 5. Gauge the likelihood of project success through opportunity assessment 

Step 6. Define participants in the needs assessment 

Step 7. Design data collection strategies 


DATA COLLECTION 


Step 8. Determine sampling scheme 

Step 9. Design and pilot data collection instrument(s) 

Step 10. Gather and record data 


DATA ANALYSIS, DATA REPORTING, AND PRIORITY SETTING 


Step 11. Perform data analysis 

Step 12. Determine priorities and identify potential solutions 

Step 13. Synthesize information and create a report 
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Planning 

Step 1. Set the focus, refine the issue(s) and identify the stakeholder(s) 

It is probably safe to assume that the desire to conduct a needs assessment did not fall 
out of the sky. Specific areas of interest may have been selected based on expectations 
and needs delineated in the agency's strategic or management plan. Research conducted 
by NOAA scientists may prompt targeting a specific issue as an education priority. 
Comments and other anecdotal evidence collected over time from facility users and 
program participants may have steered project planning in a particular direction. 
Sometimes, general trends, environmental phenomena, or public policy priorities such as 
climate change or the implementation of the No Child Left Behind Act, may suggest that 
an educational need exists. 

In this first step, those who are initiating the needs assessment specify the scope of the 
social, economic, or environmental (SEE) issues that need to be addressed ("What is") as 
well as the desired SEE conditions ("What should be"). The issues should flow directly 
from the agency mission and program priorities. Determining the extent of the problem 
or issue provides a purpose and establishes direction for the needs assessment. Agency 
documents, reports from previous education projects, census statistics, and research 
results can help in defining the scope of the problems or issues. 

Once the overall purpose of the needs assessment is established, the key stakeholders - 
the range of individuals, including audiences, potential partners, decision-makers, and 
sponsors with a vested interest in the issue or process - are identified. Permission to 
conduct the needs assessment is secured and the decision-makers who should receive the 
results of the needs assessment are identified. 

Step 2. Establish the planning team 

A basic principle behind conducting a needs assessment is recognizing that no matter 
how knowledgeable and skilled, an education coordinator does not and cannot walk 
into the project with a complete and accurate picture of the situation. Other agency staff 
members, community members, volunteers, partners - all stakeholders in the process - 
possess critical understandings that will help focus the needs assessment. Importantly, 
some of these stakeholders may also help to build credibility inside and outside the 
agency and assist the team in gaining access to target audiences. Forming a broad, yet 
manageable, planning team is essential. 

Because many planning team members will be from outside the agency and will be 
serving as volunteers, expectations and responsibilities should be established early 
on. ft should be recognized that a number of planning team members were invited 
to participate because they represent specific stakeholder groups. Consequently, they 
each bring a different set of assumptions and priorities to the table. Similarly, agency 
participation in, and sponsorship of, the needs assessment are governed by internal and 
external motivations. Given this, it is important to establish a common purpose for the 
needs assessment and to develop a broad sense of ownership in the process. 
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Why bother with stakeholders? 


“We would never have known about..is a common response when education coordinators are 
asked what they learned from their stakeholders during planning. Regardless of how well educa¬ 
tors think they know their audience, listening to stakeholders is an important task during the plan¬ 
ning and needs assessment stage. 

In one program, the overlooked stakeholder was the office director. That program ran into real 
problems when the office director revealed concerns over the focus of the program. In another 
example, the educator forgot to ask the teachers who the program had planned to train, how or if 
the topic might fit into the science standards of that grade level. Needless to say, teachers weren’t 
as excited about the program as the planner thought they’d be! 

Stakeholder involvement in planning can vary greatly. In some cases, a whole group of stakeholder 
representatives might be brought together to talk with the program planners. For example, the 
Coastal Training Program (CTP) Coordinator for the Padilla Bay National Estuarine Research Re¬ 
serve (NERR) in Washington’s Puget Sound brought together a technical advisory group consisting 
of stakeholders representing the Padilla Bay Reserve, Washington State’s Department of Ecol¬ 
ogy, Puget Sound Action Team, Washington Sea Grant, and local planning departments. The CTP 
Coordinator views her advisory group as a “think tank” that works together to identify, design, and 
distribute programs to local environmental planning and permitting professionals. In most situa¬ 
tions, however, individual stakeholders are called on the phone or spoken to in casual conversation 
about the pending program in order to get their informal feedback. Given the limited amount of 
time and staff needed to conduct a needs assessment, some project coordinators use this method 
to gauge interest and solicit feedback from stakeholders about potential projects. 

And what are stakeholders asked? Some are asked questions as simple as, “What do you think 
about this idea?” For major projects, however, stakeholder input is much more vital and therefore 
more formalized. The bottom line is, as one educator put it, “We’d have made a lot more mistakes 
if we hadn’t talked with the people who have a reason to care about the program.” 

V_____ ) 


Step 3. Draft a plan for carrying out the needs assessment 

As with any other project, the planning of a needs assessment can easily spin out of 
control. The desire to be all inclusive, both in terms of issues and audiences, can become 
overwhelming. Recognizing that the primary purpose of the needs assessment is to help the 
education coordinator design an appropriate project; a plan for how the needs assessment 
will be carried out should be developed. The planning team must determine the scope 
of the needs assessment. That is, the team must determine the type, breadth, and depth 
of information the needs assessment should be designed to gather. The plan should be 
clear and precise enough to guide further development and implementation of the needs 
assessment, specifying what data are to be collected and identifying who should be targeted 
for participation in the needs assessment. A good plan will also include a time line for data 
collection as well as a budget for carrying out the needs assessment. Keep in mind that the 
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planning team must, however, balance the need to focus the needs assessment with the 
desire to remain flexible and open to new information. 

Step 4. Use the TOP model to direct data collection efforts 

With the scope of the needs assessment established, the planning committee turns 
its attention to the TOP model. Data collection for a needs assessment occurs at the 
upper three levels of the TOP program development staircase. Comparing the desired 
or envisioned social, economic, and environmental (SEE) conditions with current or 
baseline conditions establishes the need or issue. Similarly, comparing the desired 
and baseline practices or behaviors that gave rise to the condition and the knowledge, 
attitudes, skills, and aspirations (KASA) that inform people's behaviors help identify 
project needs. The discrepancy or gap between the desired and baseline conditions is 
where project ideas stem from. 

If possible, baseline data should be gathered from existing sources, such as previous 
Program Baseline Assessments, literature review, project documents, and performance 
measures. Lack of progress in meeting a performance measure, for instance, may be 
relevant in establishing future priorities. If viable, valid, and reliable data already exist 
for a particular aspect of the needs assessment, it may not make sense to spend valuable 
resources to collect new data. On the other hand, if existing data sources are not 
available, chances are the project will be best served by collecting data from a range of 
stakeholders. 


Using TOP to collect data for a needs assessment 

Project development 



Social, economic, 

& environmental 
conditions 

How do current social, economic, and environmental conditions compare 
to the desired conditions? 

r 


Practices & 
behaviors 

What practices or behaviors must be adopted to have an 
affect on the desired SEE conditions? How do the desired 
practices compare with current practices? 
r 



Knowledge, 

What KASA levels are needed to change 



attitudes, skills, 

practices? How do desired KASA levels 



& aspirations 

r 

compare with current KASA levels? 

r 

Source: Bennett, C. and Rockwell, K. (2004). Targeting Outcomes of Programs (TOP): A hierarchy for targeting outcomes and evaluating 
their achievement. Retrieved January 2009, from University of Nebraska-Lincoln website: http://citnews.unl.edu/TOP/index.html 
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Step 5. Gauge the likelihood of project success through opportunity assessment 


While the needs assessment helps identify and determine the scope of specific conditions 
or issues, opportunity assessment gauges the probability that an agency can ameliorate 
the issues identified at the SEE, practices, and KASA levels. Opportunity assessment occurs 
at the lower four levels of TOP's project development staircase: reactions, participation, 
activities, and resources. Opportunity assessment enables an agency or organization 
to assess the likelihood of project success. At the reaction level, public responses may 
be solicited to gauge interest in potential projects that address SEE issues. Opportunity 
assessment entails estimating the reactions that would be required from the target 
audience, the scope of participation needed in various activities, and resources that will 
be required to achieve desired outcomes at the KASA, practice, and SEE levels. 

This is also a good time to inventory existing education programs and resources that 
relate to the topic or issue under consideration. Other agencies or organizations may have 
already developed an education project that can be adopted or adapted. For example, 
monitoring protocols and support materials already exist for a number of citizen 
scientist initiatives around the nation. Even if the exact procedures cannot be used in 


Using TOP to assess project alternatives and agency capabilities 

Project development 




Reactions 

How is the intended audience likely to react to program activities? What 
kind of promotional strategies are needed to attract the target audience? 

' 


Participation 

Who are the intended program participants? What is their current 
involvement with the issue compared to the desired involvement? 

' 


Activities 

What activities are currently available to the audience? 

How do these activities compare with what is needed? 

What delivery methods are currently used to convey 
content? What methods are desirable? 

' 




Resources 

What resources (time, staff, expertise, 
money) are needed to support the 
activities? What resources are currently 
available? 

' 

Source: Bennett, C. and Rockwell, K. (2004). Targeting Outcomes of Programs (TOP): A hierarchy for targeting outcomes and evaluating 
their achievement. Retrieved January 2009, from University of Nebraska-Lincoln website: http://citnews.unl.edu/TOP/index.html 
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the new situation, much can be learned from materials that others have designed and 
implemented. Similarly, existing literature can often provide important information on 
how to work with particular audiences to ensure that a project is age appropriate and 
culturally sensitive. Time spent researching may well save considerable time and effort at 
a later stage in the project planning process. 

Step 6. Define participants in the needs assessment 

In Step #2, education coordinators identified key stakeholders inside and outside the 
agency. Although many of these stakeholders were invited to participate in the planning 
team, now that a plan for the needs assessment has been developed, it is time to revisit 
and refine the list of stakeholders and potential participant groups. 

Depending on the purpose of the needs assessment, the list of key stakeholders may need 
to be expanded or reduced substantially. Decisions must be made about whom should 
be targeted for participation in the needs assessment - what type of data needs to be 
collected and from whom. For instance, if the needs assessment relates to the development 
of a citizen science project, data collection might target multiple groups, including 
NOAA scientists involved in research on the topic, teachers who might supervise student 
involvement, community service organizations that could sponsor teams of monitors, and 
state or local government agencies that might provide monitoring sites. 

It is reasonable to assume that data will be collected from groups whose characteristics 
are well known by the planning team, such as stakeholders within the agency (e.g., 

NOAA scientists) and within partner organizations. It is also reasonable to assume that 
data will be collected from groups of individuals whose make-up may be less familiar 
to the project planning team, such as members of specific community groups. Learning 
about each of the stakeholder groups is essential to the success of future steps in the 
needs assessment process. For example, what are the best methods for reaching the 
participants (e.g., phone, mail, on-line survey, during community meetings)? How 
many people belong to the stakeholder group - that is, what is the estimated size of 
the population? This information will be essential for determining the sample size (see 
Appendix B). In designing the data collection, do education coordinators need to adapt 
methods to meet the needs of a particular group (e.g., literacy rates, language)? The more 
that is known about the characteristics of each participant group, the greater the chance 
that data collection will run smoothly and wasted efforts will be minimized. 

Step 7. Design data collection strategies 

By this point in the process, the planning team identified the proposed participants in the 
needs assessment and why each group has been selected. Reflecting back on the purpose 
of the needs assessment, key questions should be developed for each participant group 
that delineates the information to be gathered at each level of the TOP programming 
staircase. From here the best data collection methods are identified. This process 
matches participant groups with key questions and appropriate instruments (see Part 
IV and Appendix A), while keeping budget, personnel, and time constraints in mind. 

The planning team must determine the best data collection method given the specific 
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situation (e.g., assessing the knowledge of adults who are not familiar with the agency). 
Keep in mind that no one data collection method fits all situations; each method comes 
with its own strengths and weaknesses (see Part IV for a discussion of data collection 
instruments). 

Before moving to the next stage (data collection) in the needs assessment process, the 
project should assess whether approvals are needed or desired, e.g., because of Paperwork 
Reduction Act regulations. Collecting data often takes the project public and requires a 
new commitment of funds and other resources. 


Data Collection 

Step 8. Determine sampling scheme 

Although it might be possible (and desirable) to collect data from all NOAA educators 
within a specific region, in all probability it will not be realistic to collect data from every 
educator in that same geographic region. Consider how difficult it is to conduct the 
U.S. Census every ten years. Even with a substantial effort, the U.S. Census is not able to 
survey everyone in the population. Unless a participant group is small and well-defined, 
collecting information from the entire group is usually not practical. Similarly, even if the 
group is reasonably small, it might not be realistic to conduct a lengthy interview with 
each and every member. 

Once the decision is made to collect data from a portion or subset of the population, 
education coordinators risk injecting bias into the process and limiting the usefulness 
of the data. Careful design of the data collection process and use of random sampling 
procedures can reduce the likelihood of this error (see Appendix B). The goal is to draw a 
sample that accurately reflects or is representative of the population as a whole. 

(D A sample is considered representative of the population if every person in the population 
has an equal chance of being selected. 

Similarly, the number of people sampled (sample size) impacts the degree of sampling 
error. Typically, sampling errors are reduced when the sample size is increased (See 
Determining Sample Size - Rules of Thumb in Appendix B). However, a larger sample, 
while reducing error, increases time and other costs. 


Step 9. Design and pilot data collection instrument(s) 

Using the key questions developed in Steps 4 and 5 as a guide, the data collection 
instruments are designed. Whether individuals will be asked to participate in a focus 
group or to complete a survey, decisions need to be made about what questions should 
be asked and how best to ask those questions. Similarly, if the decision has been made 
to observe a set of behaviors (such as how visitors interact with a hands-on exhibit), the 
specific behaviors of interest must be delineated. Each step in the procedure needs to 
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be articulated in the data collection protocol to ensure that the process is clear and well 
thought out. 

Once the data collection instrument has been designed, it should be pilot tested. What 
seems to be reasonable and clear to the planning committee may make little sense in the 
field or to individuals who are not familiar with the project. Pilot testing can highlight 
instances where directions might be misleading or meanings might be misinterpreted. 
Particularly in the case of interviews or open-ended questions, pilot testing can help 
education coordinators determine if the desired level of data will be collected given the 
questions asked, and what, if any, follow-up questions should be asked. 

Finally, the planning committee must determine that the instruments are valid and reliable. 
That is, they must ensure that the instruments are measuring what they are designed to 
measure (validity) and that the measurements are consistent (reliability). Information on 
establishing the reliability and validity of the data collection instruments can be found in 
Part IV. Data Collection Instruments. 


Step 10. Gather and record data 

Orchestrating the data gathering process requires substantial planning and day-to- 
day management. For example, contact information must be acquired, materials (e.g., 
interview schedules, surveys, return envelopes) need to be printed, focus group meetings 
and interview times must be scheduled, and arrangements must be made to tape record 
interviews and focus groups. You may find a master calendar helpful to plot each point 
in the process. The planning team will need to determine who will collect the data (i.e., 
who will conduct the interviews, who will facilitate focus groups, etc.), whether those 
collecting data need training to ensure that proper protocols are followed, and how the 
rights of study participants will be guaranteed. Not only do ethical issues arise related to 
confidentiality, but informed consent and oversight by a human subjects review board 
may be warranted. 

Once data are collected, the data will need to be recorded or entered into the appropriate 
format for analysis. Survey data will need to be entered into a database. Interviews and 
focus group sessions will need to be transcribed. As with each step in this process, care 
must be given to how the data are handled to preserve confidentiality and to ensure that 
the data are recorded accurately. 


Data Analysis, Data Reporting, and Priority Setting 

Step 11. Perform data analysis 

The type of data analysis will depend on both the instruments used and the study 
questions being answered. Some portion of the data analysis may involve a simple 
tallying of responses - numbers or percentages of those indicating yes or no. In most 
cases, however, far more sophisticated statistical analyses that examine the relationships 
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How much data should I collect? 


How much data do education coordinators actually gather as they’re planning a program? Do they 
really go through all these steps? And if they go though the steps, how much information do they 
really use? 

Not surprisingly, there is a consistent relationship between the size and/or cost of the program 
and the rigor and amount of data gathered. In planning for short-term, small-budget programs, 
many educators talk casually with only one or two people. For larger efforts involving developing 
resources, making repeated contacts, or training programs, some educators regret that they didn’t 
get more information in advance, while others acknowledge, “We could have gotten more, but 
what we got told us a lot more than we expected.” 

Here are some ways education coordinators gather data for their needs assessments: 

Telephone: 

• Call or write known stakeholders and asked them a series of questions 

• Make cold calls to people recommended by others 

Face-to-face: 

• Informal dialogues with interested people 

• Semi-structured interviews 

• Formal, structured interviews 

Mail or E-mail: 

• Mail follow-up surveys to program participants 

• Email questions to colleagues 

• Email Listserves to get additional input 

The following examples highlight methods NOAA education coordinators have used to collect data 
for their needs assessments: 

• Members of the Alaska Fisheries Science Center (AFSC) Committee on Outreach, Education, 
and Diversity developed an online survey to assess the professional development needs of 
outreach and education staff in all AFSC divisions. The results of the online survey indicated 
a need for training in networking skills, partnership development, creating lesson plans, and 
conducting project evaluations. 

• Marine education specialists from North Carolina Sea Grant conducted a mail survey of 1,135 
elementary school teachers in North Carolina to determine how teachers integrate ocean edu¬ 
cation topics in their classrooms. The results emphasized topics that teachers wanted to learn 
more about as well as their preferred delivery methods for professional development. 
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or patterns among responses may be appropriate. For qualitative data, collected most 
often through interviews and focus groups, categories and themes will need to be 
extracted from transcripts. These types of analyses will require someone with specific 
research and evaluation expertise. 

Step 12. Determine priorities and identify potential solutions 

Once the data analysis is completed, your team is finally in the position to articulate the 
results in terms of needs. Now you can systematically describe the discrepancy between 
what is and what should be. In all probability, a large number of needs will be identified 
in this process. The team must evaluate the list of identified needs and prioritize them. 

It is easy to imagine that priority setting could become a difficult (and potentially 
contentious) task for the team. 

A variety of priority-setting approaches are available to the planning team. In the end, 
however, the team must determine a set of criteria for making decisions. Sork (1995) 
recommends that needs should be evaluated against two primary sets of criteria: 
importance and feasibility. Importance can be determined by examining characteristics 
such as the number of people affected and the degree of fit with organizational or 
agency goals. Feasibility relates to characteristics such as the potential efficacy of a 
project designed to address the need. 


Sork’s Priority Setting Criteria 


Importance Criteria 

1. How many individuals are affected by this need? The greater the number affected, the greater 
the importance. 

2. If we dealt with the need, to what extent would it contribute to organizational goals? The larger 
the contribution, the greater the importance. 

3. Does the need require immediate attention? Or can attention be deferred for a period of time, 
with the need tending to resolve itself with the passage of time? 

4. How large is the discrepancy between the desired status for an area of concern and the current 
level of achievement in regard to it? The greater the discrepancy, the greater the importance. 

5. To what extent would resolving a need in this particular area have a positive effect on a need in 
another area? The more positive the effect, the greater the importance. 

Feasibility Criteria 

1. The degree to which an educational intervention, particularly an adult education strategy, can 
reduce or even eliminate the need. 

2. The extent to which resources are, or could become, available for programs to reduce the need. 

3. The commitment or willingness of the organization to change. 

Source: Altschuld, J.W. and Witkin, B.R. (2000). From needs assessment to action: Transforming needs into solution strategies. 

Thousand Oaks, CA: Sage Publishing, Inc. p. 110. 
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With the list of needs prioritized, the project team focuses its attention on solutions or 
determining which interventions will best address these priority needs while meeting 
organizational goals. As the team considers solutions to the high priority needs, it must 
also determine if an education project is the appropriate intervention. It is quite possible 
that the best strategies for addressing the need are structural or temporal. For example, a 
needs assessment might confirm that elementary school teachers are not incorporating 
ocean literacy principles in their lesson plans. The needs assessment might also 
determine that the lack of incorporation is due more to the lack of available resources 
and administrative support than a lack of knowledge. 


Step 13. Synthesize information and create a report 

By this stage in the process, the planning team has designed and implemented a 
significant endeavor. Coalitions are being built, stakeholders have been brought into 
the planning process, data has been collected, priorities have been set, and strategies 
proposed. It might be tempting to truncate this step and jump directly to project 
design, but the importance of creating a record of the needs assessment and its results is 
essential. Administrators will need a report for approving the onset of the design phase of 
the project. Those responsible for taking the results of the needs assessment and turning 
them into an effective education project will need to be able to draw from the results 


Designing Needs-Based Action Plans 


After sorting through priorities and determining potential solutions or strategies, the planning team 
will have considered numerous alternatives. Now, carefully take a second look at potential solution 
strategies. 

Revisit the most likely solution strategy and compare it to its strongest competitors. Ask the following 
key questions: 

• What features of the preferred solution strategy are its strongest assets? 

• What are some of its less strong and even weak features? 

• Were some of the competing strategies stronger in these areas? 

• Would it be possible to combine features of solution strategies without compromising the integrity of 
the top solution? 

• What is the scope of the solution in terms of location within the organization (i.e., will it reside in 
one unit or subsystem, or will it cut across many units)? 

• How much time will the solution require - that is, from a relatively short time frame to develop and 
implement (less than 6 months) to intermediate and long-range time frames (2-5 years or even longer)? 

• What is the scope of the solution in terms of focus - from narrow and fairly easy to implement to 
broad and complex to implement? 

• Is it possible to divide a large solution strategy into smaller increments for implementation? 

Source: Altschuld, J.W. and Witkin, B.R. (2000). From needs assessment to action: Transforming needs into solution strategies. Thou¬ 
sand Oaks, CA: Sage Publishing, Inc. p. 175. 
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on a regular basis. Finally, access to the results of the needs assessment is valuable as a 
foundation for future project planning efforts. So, write the report. 


Should a Consultant be Hired? 

Much of the expertise needed to conduct a needs assessment will be found within the 
planning team. If the team has been carefully crafted, members represent many of the 
stakeholder groups and can provide valuable information and insight. However, if the 
team does not include someone with significant educational research and evaluation 
experience, it may be necessary to hire a consultant. Determining a sampling scheme, 
designing data collection instruments, and analyzing the data all require specific 
knowledge and skills that cannot be substituted. Additionally, an argument can be made 
that an external consultant provides distance and objectivity that may not be possible 
for members of the planning team with vested interests. The education coordinator will 
need to assess the situation and determine if a consultant should be hired to maintain 
credibility. The box below provides guidance on working with consultants. 


Hiring a Consultant 


As with most projects, deciding to hire a consultant is only the first step. The following should help 
guide the team as you work through the process: 

• Define the scope of work. Be as specific as possible. Determine what you want to accomplish and 
what aspects of the needs assessment can be conducted by the project team. 

• Determine the budget. 

• Identify consultants with experience conducting needs assessments and evaluations. 

• Interview at least two consultants who seem most qualified. During the interview try to assess: 

Relevance of previous evaluation experience to the specific needs of the project. Has the 
consultant worked on similar projects? 

Workload - how likely is the consultant to be able to meet timelines? Will there be other 
members of the consulting team? If so, how will they divide the work? 

Work style ■ Will the consultant work with the project team as a partner, while maintaining 
objectivity? Will the consultant customize the needs assessment strategy to the project or 
merely adapt an existing one? Is the consultant willing to share his/her expertise with the 
project team so that the team learns from the experience? 

• Request a written proposal that details process, timeline, responsibilities, and budget. 

• Ask the consultant for a recent client list. Select names from this list rather than asking the 
consultant to supply two or three names of references. Use references to try to gauge the 
consultant’s ability to meet deadlines and to adapt to unforeseen circumstances. 

• Select a consultant based on qualifications and “match.” 

• Develop a written agreement. Spell out, in writing, expectations, deliverables, timeline, and 
budget. Make sure that the contract meets any applicable regulations or policies. 

• Be open and willing to change. To design and implement a useful needs assessment or evaluation, the 
consultant must understand fully the situation and any problems that the project team is facing. 
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Part I Wrap-Up 

A needs assessment helps education coordinators understand more fully the gap between 
what is and what should be. By conducting a needs assessment before the project 
is designed, assumptions integral to the development of the project (e.g., audience 
characteristics, current level of knowledge) can be confirmed or negated, avoiding costly 
missteps. The data gathered during a needs assessment can also help project designers 
define goals and objectives, and identify stakeholders and potential collaborators. This 
information will be essential as the project team turns to the next phase of the planning 
process: project design and implementation. 
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Part II. 

Project Planning and Implementation 



Introduction 

Once the need for a project is understood and authorization to proceed is secured, the 
planning team faces the task of designing a specific education project that addresses 
that need. Project planning requires an attention to what might at times seem like 
minuscule detail, while always keeping the big picture in mind. In the end, effort spent in 
planning an education project well will pay off in its ultimate effectiveness. This portion 
of the manual has been developed to help project coordinators anticipate some of that 
minuscule detail and define the project's desired outcomes - ultimately, what we hope 
participants will learn. 


Planning and Implementing an Education Project 

Just as conducting a successful needs assessment depended on taking a systematic 
approach, project design benefits from careful attention to the planning process. The time 
involved in planning an education project usually depends on the project's complexity 
and the number of stakeholders involved. 

The following 12 steps of planning and implementing an education project break 
down a complex process into manageable steps. As with each of the major sections in 
this document, please recognize that much of the detail has been left out. The 12 steps 
simply provide a generalized overview. For instance, the need to establish a budget 
and a schedule are only briefly mentioned; obviously, each requires considerable effort 
to develop. At each step, determining how extensive a process should be undertaken 
depends on the nature of the project. Obviously, a project that involves the development 
of materials for others to use will be very different from one that involves training 
individuals to make presentations to civic groups or teach school children. 

The steps outlined below should be adapted to meet the specific needs and resources 
of the project. As mentioned previously, the examples in this section can provide an 
indication of how others have planned education projects in similar situations. 
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The Project Planning and Implementation Process 

Step 1. (Re)assess need and capability 


Step 2. Establish the project planning team 


Step 3. Develop project goals and objectives 


Step 4. Develop a logic model 


Step 5. Select and characterize the audience 


Step 6. Establish program format and delivery system 


Step 7. Ensure quality instructional staff 


Step 8. Ensure quality instructional materials and strategies 


Step 9. Assemble materials, resources, and facilities 


Step 10. Plan for emergencies 


Step 11. Promote, market, and disseminate project 


Step 12. Implement project 


Step 1. (Re)assess agency need and capability 

During the needs assessment phase the project team used the Targeting Outcomes 
of Programs (TOP) model to prioritize needs and assess opportunities. The needs 
assessment highlighted the gap between the current and desired social, economic, and 
environmental (SEE) condition as well as the practices that gave rise to the condition and 
the knowledge, attitudes, skills, and aspirations (KASA) that inform practice. With the 
needs assessment recommendations in hand, project managers should revisit the issue to 
determine if the proposed project is consistent with NOAA's mission and priorities. The 
need may have a high priority; however, it is quite possible that NOAA may not be the 
most appropriate organization to tackle it. 

The needs assessment team also inventoried existing projects based on a tentative 
understanding of the issue being analyzed. Now that the project need and potential 
strategies have been articulated further, it is important to more fully inventory existing 
projects both inside and outside NOAA to determine areas of potential duplication. 

By conducting this inventory, the interrelationships among existing projects and the 
proposed project can be considered, and the function of the proposed new project 
contrasted with existing activities. 

Similarly, project managers should take this opportunity to step back and inventory 
existing resources (human, financial, physical, and material resources) available to 
NOAA. Does the agency have the means to support the project at this time? There is little 
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to gain from entering into an extensive project design process only to determine that 
funding and personnel are insufficient to support the activity. 


Step 2. Establish the project planning team 

In all likelihood, a number of potential partnerships and collaborative arrangements were 
identified and built during the needs assessment phase. Stakeholders from both inside 
and outside of NOAA were identified. Education coordinators might want to evaluate 
how well the needs assessment planning team functioned and consider asking some or 
all of its members to continue to participate in the project planning process. Establishing 
a broad-based project planning team is one way of tapping expertise and continuing 
the process of developing partnerships and collaborations. Since the focus during this 
phase of project development is on design and implementation, it is quite possible 
that additional members will need to be recruited to the team. Depending on levels of 
expertise, individuals with an understanding of the NOAA science related to the specific 
issue as well as those with experience designing instructional activities may be needed. As 
with the needs assessment planning team, roles and expectations must be delineated. 

It should be noted that an evaluation planning team will also need to be established 
(see Part III, Step 2). To ensure that evaluation is considered at every step in the project 
design and implementation process, education coordinators should form the evaluation 
planning team now. Depending on the project, its timeline and level of complexity, the 
evaluation planning team might function well as a subcommittee of the overall project 
planning team. The importance of forming an evaluation planning team early cannot be 
overemphasized. Even though evaluation is discussed in detail later, its success depends 
upon being integrated throughout the planning, design, and implementation process. 

Step 3. Develop project goals and objectives 2 

As with any activity, it is important to know where you are going. While the needs 
assessment provided a purpose and direction for the project, planning helps fill in the 
details for how you are going to accomplish your goals and objectives. It is now time 
to revisit the project goals and objectives or targets identified at each level of the TOP 
programming staircase as these will steer much of the team's future work. At the SEE level, 
goals provide the "big picture" of what is to be accomplished by undertaking this project. 
Although project goals are typically not measurable, they describe the desired impact 
of the project in broad terms. The reason a goal is difficult or impossible to measure is 
because it is often not specific in its end point. For example, the goal of the Padilla Bay 
National Estuarine Research Reserve's Coastal Training Program (CTP) is to help coastal 
managers gain a better understanding of environmental issues and regulations. The CTP 
accomplishes this through a series of hands-on, science-based trainings for professionals 
who make decisions about coastal management in Washington State. 


2 Note that objectives and outcomes are both statements of the impacts of the project on the audience and, in the longer 
term, on the issue. Objectives are statements of the intended impacts before the project is initiated (in planning the proj¬ 
ect). Outcomes are these same impacts during or after the project is complete. 
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Objectives, on the other hand, provide specific end points or measurable outcomes and 
should be developed for each level of the TOP programming staircase. One objective 
for the CTP training course "How to determine the ordinary high water mark?" is for 
shoreline planners, responsible for implementing the Washington State Shoreline 
Management Act (SMA), to follow the proper protocol used in determining the 
Ordinary High Water Mark (OHWM) when approving shoreline development permits. 
Consequently, well-articulated goals and objectives are both essential at this stage in the 
planning process. 

Objectives describe what the specific impact of the program will be and the degree to 
which that impact must occur. Objectives also specify what the audience will be able to 
do or what the specific change will be to the resource/issue after the intervention (see 
Writing SMART Objectives for the Project on the next page). Clear, measurable objectives 
will not only facilitate the design and implementation of the project, but its evaluation 
too (see Part III). In drafting the objectives, the project team should also develop a clear 
conception of how NOAA science will be used and promoted throughout the project. 


Writing Project Goals 


A goal provides the “big picture” or the ultimate vision of the program. When writing goals ask, 

“What will this situation look like in five or ten years?” or “What is the perfect world with regard to 
this social, economic, or environmental issue or problem?” Although there are many methods of 
writing good goals, here are three suggestions for writing effective project goal statements: 

1. State the goal in the present tense (even though the goal might not be reached until five years 
from now). 

Example: “Coastal zone managers have access to and use remote sensing and geographic 
information systems (GIS) in their decision making.” 

2. Write the goal with an unspecified or indefinite endpoint. 

Example: “... to improve the ability of extension and education professionals to measure the 
impacts and outcomes of their projects and products.” 

3. State the broad anticipated impact of the project. 

Example: “.. .species diversity in the restored area is increased.” 

V_ J 

Step 4. Develop a logic model 

Once the team members have agreed on the goals and objectives, they are in the position 
to draft the rest of the project. At this stage, project planners find the development 
of a project's logic model particularly helpful. Whereas the TOP model provided the 
framework by which objectives or targets were specified, a logic model will convey the 
detail needed to carry out the program. The logic model will also serve as a guide for 
evaluation (see Appendix C for an example). A logic model is a schematic of how the 
project will work, linking project outcomes (short-term, intermediate, and long-term) 

3 Inputs and outputs in a logic model are referred to as "capacities" in NOAA's Planning, Programming, Budgeting and Execu¬ 
tion System. Input capacities may be funding, personnel, or other resources needed to do the project. Output capacities could be 
numbers of workshops held, number of target audience members in attendance, numbers of teacher resource kits produced, etc. 
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Writing SMART Objectives for the Project 

In writing meaningful objectives, many education coordinators have found a set of criteria, 
summarized by the acronym SMART, to be helpful. A SMART objective is: 

Specific: Describes an action, behavior, outcome, or achievement that is observable, (e.g., fol¬ 
low Department of Ecology’s protocol in determining the ordinary high water mark; volunteer in 
community shoreline cleanups; incorporate educational materials on aquatic invasive species). 

Action words also serve to group the objectives into specific learning domains. 


Examples of Action Words Used to Help Set Objectives for Different Levels of Learning 


Know 

Comprehend 

Apply 

Analyze 

Synthesize 

Evaluate 

define 

discuss 

demonstrate 

distinguish 

design 

appraise 

record 

explain 

employ 

debate 

construct 

assess 

list 

differentiate 

illustrate 

calculate 

create 

judge 

name 

identify 

translate 

diagram 

propose 

predict 


Measurable: Details quantifiable indicator(s) of progress towards meeting the goal 
(e.g., all local shoreline planners responsible for implementing Washington State Shoreline Man¬ 
agement Act, 70% of participants, identify five or more aquatic invasive species). 

Audience: Identifies the audience (e.g., local shoreline planners responsible for implementing 
Washington State Shoreline Management Act, workshop participants, community members) and 
describes outcomes from the perspective of the audience (i.e., what the audience will be able to do). 

Relevant: Is meaningful, realistic, and ambitious; the audience can (given the appropriate tools, 
knowledge, skills, authority, resources) accomplish the task or make the specified impact. 

Time-bound: Delineates a specific time frame (e.g., six months after participating in the 
Ordinary High Water Mark class, at the conclusion of the workshop, three months after receiving 
outreach materials). 


with project outputs, and inputs (resources). 3 Logic models provide a road map, showing 
how the project is expected to work, the logical order of activities, and how the desired 
outcomes will be achieved. 

The use of a logic model forces a "backward" design of the overall project. By first 
determining the ultimate or desired long-term, intermediate, and short-term outcomes 
(based on the needs assessment), the planning team is in a much better position to 
identify appropriate project activities, resource needs, and timeframes. 


Part II. Project Planning and Implementation 


29 












Project Logic Model 


Program Development 


Inputs 

(What we 
invest) 


Outputs 

Activities 

Participants 

(What we 

(Who we 

do) 

reach) 


Short-term 

Outcomes 

(Knowledge, 
attitudes, skills, 
& aspirations) 


Intermediate 

Outcomes 

(Practices & 
behaviors) 


Long-term 

Outcomes 

(Social, economic, 
& environmental 
conditions) 


Program Evaluation 


How logical is a program? 


Educational programs in many agencies have traditionally been designed based on the intuition and 
experience of the agency educator in charge. This does not mean the programs are guaranteed to 
either succeed or fail. It is interesting, however, to ask various educators about the steps they went 
through to design their program. In talking through how they developed the program, they often real¬ 
ize that there were gaps in their thinking. 

One purpose of program logic is to ensure that no important steps are skipped when planning a 
program. Some agencies have used logic modeling for many years and are proficient in their use 
- although, as one educator said, “we should be proficient, but we often use [logic models] to 
rationalize rather than plan a program.” Other agencies and organizations are just beginning to think 
through how program outcomes are tied to program inputs and resources. 

One educator who uses logic models describes the process as “taking a look at the components of 
your program [resources, activities, audiences, outcomes] and seeing how it’s organized.” Another 
said that doing a logic model creates “a visualization of the process. It’s something tangible that 
you can see.” One agency-based educator observed that logic models were good for “accountabil¬ 
ity—you have something tangible to take to supervisors, funders, taxpayers, etc” to show exactly 
how and why a program will work. 

Perhaps program logic can be best understood, in the words of one educator, as “the process of 
thinking through everything. It breaks the mold, stimulates new ideas, and helps clarify and crystal¬ 
lize thinking.” 
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Spotlight on Hazardous Weather Awareness Week (HWAW) 


Each year, Florida’s HWAW strives to promote hazardous weather preparedness across the state. 

The HWAW program encourages youth and adults to develop an emergency plan, restock emergency 
supplies, purchase a NOAA Alert Radio, and to act safely when threatening weather approaches their 
community. Below is an abbreviated list of the targets education coordinators plan for during HAWA. 



Improve quality of hazardous weather 
preparedness and responses 
Increase number of families equipped to 
handle hazardous weather events 

Increase knowledge of safety rules in severe 
weather 

Teachers incorporate HWAW activities into 
their curricula 

Maintain and update HWAW website 
Distribute weather guides to elementary teachers 
Organize and promote middle school HWAW 
poster contest 

Organize and promote high school HWAW 
essay contest 

Funding • Staff time 

Partners • Materials 


Long-Term Outcomes: Describe the intended ultimate impacts of the project on the issue. These 
might be social, economic, or environmental conditions. These consequences are expected to oc¬ 
cur after a certain number of behavioral changes have been made. 


Intermediate Outcomes: Describe expected impacts on the audience’s behavior because of the 
project. What behavioral changes are individuals expected to make because of the project? These 
outcomes tend to occur after there has been an earlier change of knowledge, attitudes, skills, or 
aspirations. 

Short-Term Outcomes Describe the expected immediate impacts of the project (e.g., audience 
reactions and changes in knowledge, attitudes, skills, or aspirations immediately following partici¬ 
pation in the project). 

Outputs: Describe the activities, events, products, and services that reach people targeted by the 
project. 

Inputs: List the time, money, human resources, office space, utilities, equipment, supplies, man¬ 
agement and partner support, etc. needed to accomplish the project. 


Source: Florida Division of Emergency Management (2009). Hazardous Weather: A Florida Guide. Retrieved January 2009, from 
Florida Division of Emergency Management website: http://www.floridadisaster.org/KIDS/index2.htm 
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Step 5. Select and characterize the audience 

Each audience is distinct. Not only do audiences vary in terms of background and cultural 
characteristics, personal experiences, and prior knowledge, but they also vary in terms 
of age and developmental level. Participants also vary in their motivations for learning 
depending on whether they are participating as students in a mandated, formal classroom, 
as members of a family outing, or as employees. Each of these factors is important to 
consider in determining what particular audience(s) should be targeted by the project. 


Who should the program’s audience be? 


Most NOAA education and outreach programs are targeted either to schools or to community groups. 

Yet there are many other potential audiences for programs. The decision about who the audience of a 
program is should be based in part on the needs assessment and in part on the needs of the agency. 

By far, most educational programs focus on what we call “intact groups.” These are usually school 
groups, but can also include other youth groups - Scouts, 4-H, Boys and Girls Clubs, church groups - 
and community groups; service, fraternal, professional, and interest groups. Even within intact groups, 
the vast majority of educational programs conducted by agencies are for formal education audiences. 

A mistake made by many managers and staff during development is to forget that the program is 
really for the audience and not for the agency. There are plenty of examples of programs that were 
created for school groups; then, when budgets got tight, the school groups no longer made the trip 
to the site. As long as there was funding, teachers would bring their classes, but with funding cuts, 
teachers selected the field trips that most closely aligned with their curriculum. Because managers 
had not taken the audience into account when building the program or considering the curriculum 
needs of a particular grade, the program was easily cut since teachers viewed it as an “add on” 
rather than as “core.” 

For many programs, school groups are an appropriate target audience, but the real audience is the 
teachers, not the students. Developing teacher-training programs can be a resource-effective way to 
ultimately reach more students. To reach students in grades 4 through 10, Nab the Aquatic Invader! 

(a project of the National Sea Grant College Program) trained a cadre of some 200 teachers about 
aquatic invasive species topics and how to use the Nab the Aquatic Invader! interactive website in 
their classrooms. In turn, it is estimated that these teachers reach upwards of 30,000 students each 
year. But unless the program is designed in partnership with the targeted teachers (topic and grade 
level), participation and continued use of a program are limited. 

Some education coordinators have looked for new and different audiences in order to broaden the 
reach of their agency or division. Some have targeted intact groups in different neighborhoods, while 
others have created partnerships with agencies and organizations that directly serve the audiences 
they’re trying to reach. Regardless of the program, though, the most successful educational efforts 
are those where the agency’s mission drives the educational program, the program is tied to an 
important audience of the agency or division, and it meets the needs of the target audience(s). 
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Ultimately, the project planning team will need to examine the stated outcomes and 
weigh them against the feasibility of reaching particular audiences effectively. 

Even though the audience was analyzed as part of the needs assessment, it may be 
necessary to conduct a more specific, targeted assessment now that the project goals, 
objectives, and outcomes have been identified. An audience assessment will help the 
planning team decide how a specific project can be customized to meet participant 
interests and learning styles. Some of the issues to be addressed in the audience 
assessment include: 

• Knowledge and interests: How much do they know about the topic or issue? Are they 
familiar with terminology or basic concepts? How interested are they in the topic or 
issue? To what extent are they interested in learning about the topic? 


Culturally Competent Informal Learning 


Each culture has a shared set of meanings and beliefs, and distinguishes itself from others by the 

approaches it takes toward basic issues of interpersonal and intergroup relations, time, and nature. 

Cultures typically fall in different places along the spectrum for each issue. For example: 

Interpersonal Relationships 

• Universalism versus Particularism - the importance of rules, codes, and laws versus excep¬ 
tions, special circumstances, and personal relationships. 

• Individualism versus Collectivism - the importance of the individual and personal freedom 
versus the importance of the group and cooperative and harmonious relations. 

• Neutral versus Emotional - the range of feelings expressed, whether interactions are detached 
or express emotion. 

• Specific versus Diffuse - the degree of personal involvement in transactions between people, 
i.e., whether limited to the specific aspect of a transaction or requiring a relationship with the 
whole person. 

• Achievement versus Ascription - whether status is accorded by what a person has achieved or 
by who the person is and the person’s family or social connections. 

Attitudes about Time 

• Sequential or Synchronic - how the past, present, and future relate to each other, and which 
has greatest importance; whether time is considered as a sequence passing in a straight line or 
more as moving in a circle. Standards of punctuality can range from minutes to a day or more. 

Attitudes about the Natural Environment 

• Whether the world is considered as more powerful than the individual or the individual is the 
source of vice and virtue; whether society should be considered subordinate to, in harmony 
with, or having mastery over, nature. 

Source: White, R. (2002). The importance of cultural competence to informal learning attractions. The Informal Learning Review, 

(52), 18-20. 
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• Prior educational experiences Have they participated in other educational activities 
related to the issue? 

• Extrinsic and intrinsic motivations: Why would they want to learn about the topic or issue? 

• Attitudes and preconceptions: What are their basic attitudes toward the issue, NOAA, 
education, etc.? 

• Constraints related to attending an education activity: When is the audience able to 
attend an educational event? Do they have the necessary transportation available to 
travel to a particular site? 

• Cultural characteristics: What are the audience's attitudes toward nature, society, 
learning, and other values relevant to the project? 

Answers to each of these questions will impact the design of the education project and 
its budget. 

Step 6. Establish program format and delivery system 

Given the range of alternatives available, the planning team must determine what type 
of educational activity best meets the needs of the audience and addresses the ultimate 
outcomes, budget, schedule, etc. That is, what types of activities are most appropriate - 
workshop, demonstration area, community forum, field day, course, guest speaker, etc.? 
Keep in mind that your needs assessment can provide information regarding the types 
of delivery systems your audience will accept and respond to and what content they are 
interested in. For example, a needs assessment conducted by North Carolina Sea Grant 
of the teaching needs of elementary teachers in North Carolina revealed the topics these 
teachers were most interested in learning more about, the barriers to teaching ocean 
literacy principles, and the kinds of training and materials teachers prefer when exploring 
professional development opportunities. 

In many ways, decisions concerning project outcomes and target audience, along with 
budget and time constraints, have narrowed the range of feasible options available to the 
planning team. Matching the ultimate impact or outcomes of the project with appropriate 
educational activities is essential. A complex long-term outcome that requires participants 
to develop a high level of understanding and skill will not, in all likelihood, be met 
through a two-hour workshop. Similarly, access to an audience may be limited by external 
constraints. For instance, teachers may be willing to invite outside experts into their 
classrooms on an occasional basis as long as the activity addresses educational standards. It 
would be a rare situation, however, where outside experts are given the opportunity to teach 
an entire unit or semester-long course during regular school hours. 

In selecting the appropriate scope and format for the project, planning team members 
should also take this opportunity to consider how meaningful partnerships and 
collaboration can be enhanced through the project. It is quite possible that a more 
effective and efficient long-term educational strategy can be designed by partnering with a 
community organization rather than developing a stand-alone event. 
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Step 7. Ensure quality instructional staff 

No matter what the ultimate format (course, field day, or workshop) or length of time 
(one hour, one week, or two years), someone (staff members or volunteers) will need 
to deliver the educational content through some form of instruction. The efficacy of the 
project depends on the instructional staff and their abilities. Although the educational 
materials or activity guide may give instructors direction, teaching requires skill and 
conscious effort. Instructors need preparation. At a minimum, they will require training 
in how to deliver the specific activities to the specific audience. 




How do we staff the program? 


It’s the best of all possible problems: What happens if your program is a success? In one case, a 
new program was so successful and popular that the demand quickly grew beyond the ability of 
the staff to provide access to all groups wanting to participate. In several cases, educators realized 
that quality control suffered when a program grew, and they had to hire seasonal educators to 
implement the program. 

Making good staffing decisions is part of the reason for doing a needs assessment. One educa¬ 
tor commented that through asking the potential audience about the demand for a program, “We 
knew it would be popular, and that’s why we limited our promotion of the program.” Many others 
reflected that they hadn’t planned on the time it would take to train and oversee volunteers to en¬ 
sure that the program was conducted the way it was designed. Some programs only allow certain 
staff to do certain types of educational programs, such as public speaking programs, in order to 
guarantee that the program is conducted correctly. All these are staffing decisions that need to be 
considered in planning and implementation. 

Another lesson many educators learned is that the amount of time needed for a program must 
include preparation and follow-up. A couple of typical realizations were, “We didn’t build in the 
time it took to gather up the materials before each program,” and “We didn’t build in time for doing 
a careful evaluation.” Several programs included some data collection for evaluation, but failed to 
consider the time needed to enter the data for analysis. There are other programs where analysis 
means quickly looking through the feedback forms, but never really analyzing the data or examin¬ 
ing data across programs. The reason? “We didn’t think about how much time it would take. 

Again, not all programs are created equal; different types of programs require different amounts 
of time and dedication. One lesson several educators shared is that if consistency in a particular 
program is a goal, then training and oversight are important and should be built into the program. 
Another lesson from a program suggests that the educator needs to consider the knowledge of the 
audience when making decisions about staff time. When asked why, the response was that if the 
audience knows a lot about the subject, “the educator needs to be better prepared and needs to 
spend more time staying current.” 

Ensuring quality instructional staff turns out to be more complex than it first appears! 

V_ J 


Part II. Project Planning and Implementation 


35 





The amount of time and effort expended on training instructors will depend, at least to 
some extent, on the educational activity. Short, discrete activities being led by experienced 
instructors may require little orientation. On the other hand, if individuals are involved 
in long-term educational activities, the project planning team may wish to consider a 
more extensive professional development program that includes appropriate education 
theory, instructional practices, and use of different instructional settings. 


Step 8. Ensure quality instructional materials and strategies 

Depending on the topic, audience, format, and time frame, project planners will need to 
adopt or adapt existing education materials for their particular use, or design new materials 
and strategies. In either case, the quality and appropriateness of the instructional materials 
are critical. The good news is that literally thousands of instructional materials are available. 
Lesson plans and other support materials can be retrieved easily online or accessed through 
government agencies, non-profit organizations, and commercial publishers. These materials 
and lesson plans often provide background information, outline learner objectives, and 
offer step-by-step instructions on how to teach the lessons. 

The bad news is also that literally thousands of instructional materials are available! 
Finding good quality materials that meet a specific set of needs may be time consuming 
and frustrating. On the other hand, designing curriculum materials anew is not a simple 
task and can take months. 

Although there may not be an easy way of identifying or creating quality materials, 
criteria are available to assist the planning team in the process. For instance, the 
Environmental Education Materials: Guidelines for Excellence developed by the North 
American Association for Environmental Education (NAAEE) offers recommendations for 
developing and selecting quality environmental education materials. Whether developed 
specifically for the project or adapted from existing sources, instructional materials 
should be accurate, balanced, fair, and reflective of good pedagogy (e.g., relevant 
to the learners, encouraging hands-on learning, accommodating different learning 
styles, age/audience appropriate). For projects aimed at children, it may be important 
for the instructional materials to be aligned to local, state, and national standards or 
organizational award programs (e.g., scouting badges). It is particularly important that 
any instructional materials be field tested with the target audience and under similar 
circumstances. Materials that work well in a classroom with tenth graders may not work 
at all in a field setting with young children. Similarly, materials designed for children may 
not work with adults unless they have been modified significantly. 
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Environmental Education Materials: Guidelines for Excellence 


The Guidelines for Excellence developed by the North American Association for Environmental Edu¬ 
cation (NAAEE) point out six key characteristics of high quality education materials. The Guidelines 
were designed to help educators, administrators, curriculum designers, and materials developers 
evaluate the quality of materials. 

Key Characteristic #1: Fairness and Accuracy 

1.1 Factual accuracy 

1.2 Balanced presentation of differing viewpoints and theories 

1.3 Openness to inquiry 

1.4 Reflection of diversity 

Key Characteristic #2: Depth 

2.1 Awareness 

2.2 Focus on concepts 

2.3 Concepts in context 

2.4 Attention to different scales 

Key Characteristic #3: Emphasis on Skills Building 

3.2 Critical and creative thinking 

3.3 Applying skills to issues 

3.4 Action skills 

Key Characteristic #4: Action Orientation 

4.1 Sense of personal stake and responsibility 

4.2 Self-efficacy 

Key Characteristic #5: Instructional Soundness 

5.2 Learner-centered instruction 

5.3 Different ways of learning 

5.4 Connection to learners’ everyday lives 

5.5 Expanded learning environment 

5.6 Interdisciplinary 

5.7 Goals and objectives 

5.8 Appropriateness for specific learning settings 

5.9 Assessment 

Key Characteristic #6: Usability 

6.1 Clarity and logic 

6.2 Easy to use 

6.3 Long-lived 

6.4 Adaptable 

6.5 Accompanied by instruction and support 

6.6 Make substantiated claims 

6.7 Fit with national, state, or local requirements 

Source: North American Association for Environmental Education (1996, 2004). Environmental Education Materials: Guidelines for 
Excellence. Washington, DC. Retrieved December 2003, from NAAEE website: http://www.naaee.org/programs-and-initiatives 
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Developing quality instructional materials and strategies represents a significant process 
in and of itself. Time, effort, and expertise are needed. Since instruction is the center of an 
education project, this step cannot be abbreviated. Although there are many approaches 
to instructional design, some find the five-step ADDIE model particularly helpful: 


1. Analyze: Focuses on understanding the audience and what they need to learn, determining the budget, 

identifying constraints, and setting a time line. 


Focuses on selecting the specific subject matter, writing instructional or learner objectives, 
determining appropriate sequencing of subject matter or skills, and developing instructional 
strategies. 


3. Development: Focuses on the creation or selection of instructional materials (including detailed lesson plans and 
other resources such as websites) that support instructional objectives. Field-test materials and 
revise if necessary. 


4. Implement: Present instructional activities. 


5. Evaluate: Assess the success of the instructional design. 


It should be noted that project planners write instructional or learner objectives as part 
of the materials development process. Like the project objectives developed in Step 3, 
SMART objectives should be developed for each instructional activity that describes 
learner behavior, attitudes, knowledge, and/or skills as a result of instruction. These 
objectives describe the intended impacts or results of the project on participants and/ 
or the issue (How will the participants change? or How will the current situation change 
with regard to the issue?) rather than the process of instruction itself. Instructional 
objectives support project objectives and should lead directly to the desired short-term, 
intermediate, and long-term outcomes expressed in the project logic model. 


Step 9. Assemble materials, resources, and facilities 

Deficiencies in logistical planning - facilities, materials, and equipment - can sink even 
the best instructional design delivered by the best educator. The importance of this step 
cannot be underestimated. More often than not, miscommunication and assumptions 
lead to logistical difficulties during project implementation. Activities planned for a 
single class of 30 sixth graders may turn chaotic if there are three classes or 90 students 
instead. Room arrangements (e.g., theatre style versus round tables with moveable chairs) 
can make a significant difference depending on the planned presentation type. Planners 
must think ahead to the actual implementation of the project and anticipate a variety of 
needs, including: 

• materials (e.g., newsprint and markers) 

• equipment (e.g., computers, overhead project, monitoring equipment) 

• comfort and amenities (e.g., food, restrooms, heating and cooling) 

• facilities (e.g., indoor and outdoor spaces, ambience, seating arrangements). 
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Spotlight on Nab the Aquatic Invader! 
Be a Sea Grant Super Sleuth 


The goal of Nab the Aquatic Invader! (NAI) is to prevent new introductions and the further spread 
of aquatic invasive species by providing information to students and teachers in grades 4-10. 

Through an online educational experience, NAI allows them to better understand aquatic invasive 
species biology, impacts, and control measures so they can make wise decisions and take ap¬ 
propriate actions. 

Developing NAI was no small task! Education coordinators from Indiana-lllinois, New York, Or¬ 
egon, Louisiana, and Connecticut Sea Grant recruited teachers to help develop online curriculum 
materials and serve as mentors to other educators. Part of the teachers’ time was spent becoming 
familiar with regional aquatic invasive species that were then used as their foci for writing new 
descriptions and online activities. Teachers were responsible for aligning the activities to both state 
and national content standards and were encouraged to include activities that cut across science, 
math, language arts, geography, and social studies. The teachers’ field tested the activities in their 
own classrooms and evaluated their success. 

In evaluating the design, development, and implementation of the NAI materials education coordi¬ 
nators were interested in the following questions: 

• Are the new activities effective in conveying concepts about introduced species? 

• is the format and design of each activity practical for classroom instruction? 

• Are the online activities effective in encouraging a new sense of responsibility and steward¬ 
ship? 

• Do the teachers plan to continue networking with one another beyond the project’s duration? 

• What types of stewardship activities were conducted by the students after participating in NAI? 

Once the evaluation was complete, teachers then served as Sea Grant education ambassadors by 
sharing their new knowledge and messages for preventing the spread of aquatic invasive species 
at in-service teacher workshops and state and national professional education conferences. The 
workshops serve two purposes: 1) to introduce and show how the aquatic invasive species materi¬ 
als and activities available through the NAI website can be incorporated into the classroom and 
2) to demonstrate how teachers can help their students develop community stewardship projects 
focused on some aspect of the aquatic invasive issues. 

Source: Goettel and Domske. Nab the Aquatic Invader! Be a Sea Grant Super Sleuth. Retrieved January 2009, from Sea Grant 
Nonindigenous Species Site (SGNIS) website: http://www.sgnis.org 
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A number of other details also need to be considered: coordinating schedules 
with collaborators and partners, double-checking participant lists, and confirming 
participation of honored guests. Communication, checklists, and clearly delineated roles 
can limit disruptions. 


Step 10. Plan for emergencies 

The old adage, "What can go wrong, will go wrong," is just as applicable when designing 
an education project as with any other activity. Education coordinators must plan for 
emergencies and the inevitable mishap. Especially if participants will be outdoors, engaging 
in physically challenging activities, or in the proximity of potentially dangerous materials, 
emergency preparations are a must. Leaders must know what to do in the case of severe 
weather or injuries. They need to know whom to contact for medical assistance and how 
to report an emergency such as fire. Education coordinators need to develop a system to 
warn leaders of severe weather or other emergency situations. Even though the worst rarely 
happens, planning for it will provide those involved with a plan that was devised during 
calmer times and will prevent the emergency from becoming worse. Promotional materials 
should indicate relevant information about participation in the activities, including level of 
physical activity, appropriate clothing, safety concerns, etc. Similarly, liability concerns and 
any regulations need to be taken into consideration during planning. 


Step 11. Promote, market, and disseminate the project 

Although some projects flow into regularly scheduled activities with a "captured" 
audience, most projects depend on some form of promotion or recruitment for 
participation. Even though an education activity geared toward school children does not 
need to recruit the children as participants, their teachers do need to be reached. Visitors 
to a wetland area on a Saturday afternoon need to be made aware that an education 
activity is available and, importantly, they need to see why participating in that activity 
would be worth their time. 

Few projects have the luxury of a built-in audience. Most projects need to plan a 
promotion and marketing strategy and secure sufficient resources (including time) to 
implement the plan. The key is to plan a marketing program that reaches the target 
audience and motivates them to participate. Partners, media, websites, and word of 
mouth may all be key to successful promotion. In addition, project planners should 
not overlook participants in current education programs as potential recruits and 
ambassadors. It should be remembered, however, that promotion and marketing require 
expertise. If this expertise is not available in-house or within the planning team, hiring a 
consultant may be necessary to ensure success. 

The project planning process depends upon a great deal of forward thinking. The 
planning team should begin to think about steps beyond the actual implementation and 
evaluation of the project. Assuming that the project is successful, planners should begin 
to consider how to disseminate project products. Building in a dissemination plan helps 
to close the loop on project development and ensures that someone else does not waste 
time and effort reinventing the wheel. 
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If you build it, will they come? 


One very surprising finding from interviews of agency educators is their sense that if a program 
is created, there will be an audience. Some managers base this assumption on their needs 
assessment, but in general, there was a belief that people would naturally be interested in the 
program and would show up. Very little attention was paid to how these people would find out 
about the program. In other words, there wasn’t much thought given to promotion. And as one 
educator noted: “Sometimes we make a product and don’t go to the intended audience to see if 
it’s what they need.” 

In some situations, the educator used a newsletter to inform people about the program. In 
one such situation, the program was designed to reach a new audience. However, no one had 
thought in advance about the fact that a newsletter sent to previous participants would not 
reach a new audience. 

Similarly, new on-site programs advertised by signs and handouts will not reach a different 
audience than the ones that already come. Several educators believe “word of mouth” is the most 
effective way of reaching their desired audience, but others have discovered the need to “reach 
some of the desired audience first in order to get the word-of-mouth going.” Many educators who 
focus on teacher training used word-of-mouth effectively because they were able to reach their 
core teachers, who did pass on the word to their colleagues. Attempts to reach new audiences 
posed the biggest challenges. 

A few educators mentioned that they changed their programs once they were implemented, based 
on who was attending. Rather than marketing toward their original audience, they decided that 
those attending were an appropriate audience for their programs. This was often the case for 
programs with small resource bases. In a couple of situations, a concerted effort was made to 
“create bridges to the community” in order to reach the desired audience after the program was 
already in place. 

It turns out that building and implementing a thoughtful plan for marketing or promotion is an 
important and often overlooked step. 

V_ ) 


Step 12. Implement the project 

The day will come when the project is actually implemented. All of the planning effort 
is finally realized. Each type of event (e.g., workshop, class meeting, community forum, 
field day) requires a unique set of actions, materials, and equipment. Hopefully, adequate 
pre-planning will have been done, allowing instructors and leaders to focus on the needs 
and educational experience of the participants. Event planners often find that creating 
checklists and detailed agenda are helpful. The more complex the event, the more 
important it is that each person knows what to do when. 


Part II. Project Planning and Implementation 


41 





Part II Wrap Up 

Project design and implementation are the heart of an education program. As this 
section has underscored, careful attention to the details of planning will pay off in 
program results. Use of a logic model can be invaluable in setting clear expectations and 
seeing how all components of the program work together. Attention to the details of 
implementation is also essential, including planning for emergencies and contingencies. 
Finally, promotion and marketing of an education project should receive systematic 
attention in the planning phase. All of the key elements of the program should be 
monitored during the implementation phase to ensure that they are running smoothly 
and to undertake mid-course corrections as needed. 
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Part III. 

Project Evaluation 



Introduction 

To get to this point in the project design process, a considerable amount of time, effort, 
and other resources have been expended. Quite obviously, the goal is to create effective 
education projects that can serve as models of excellence. But how do you know if you 
have succeeded? 

This portion of the guide has been developed to help education coordinators take project 
development to a new level by truly integrating evaluation into the process. Part III walks 
through the basics of evaluation, outlining everything from types of evaluations and ways of 
collecting information to the use of outside evaluators and ethical considerations in gathering 
data from program participants. This information is intended to answer questions about 
project evaluation and provide guidance in using evaluation as a project improvement tool. 


What is Project Evaluation? 

In the course of implementing a project various types of information are gathered. 
Education coordinators often want to know how many individuals participated in an 
event, whether participants were satisfied with the logistics, or whether staff members and 
volunteers feel confident in their ability to deliver a particular educational experience. 
Answers to these questions provide useful information. They help education coordinators 
monitor specific aspects of the project. However, in practice, this type of information 
gathering tends to be more sporadic and patchy than methodical and comprehensive. 

Evaluation is the systematic collection of information about activities, characteristics, 
and outcomes of projects in order to make judgments about the project, improve 
effectiveness, and/or inform decisions about future programming (adapted from Patton, 
2002). Importantly, evaluation provides project coordinators with well-documented and 
considered evidence to support the decision-making process. Evaluation is not merely the 
accumulation and summary of data and information about a project. 

Project evaluation helps determine a project's merit (does it work?) and its worth (do we 
need it?). Evaluation helps decision-makers determine if a project should be continued 
and, if so, suggests ways to improve it. Additionally, evaluation documents project 
(and program) accomplishments. If the project has been designed properly with well- 
articulated objectives that specify what must be accomplished, to what degree, and within 
what time period, the evaluation can determine whether or not the objectives have been 
met. The evaluation can also gather information as to the reasons why a project is or is 
not meeting its objectives. 
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Spotlight on B-WET 
(Bay Watershed Education & Training) 


NOAA B-WET is an environmental education program that supports experiential learning through 
local competitive grants. Currently B-WET Programs are implemented in the Chesapeake Bay 
(MD, VA, and DE), California, and the Hawaiian Islands. The goal of B-WET is to support existing 
programs, foster new programs, and encourage partnerships in providing “meaningful watershed 
environmental experiences” (MWEEs) to students as well as professional development opportuni¬ 
ties for teachers in utilizing MWEEs in their own classrooms. 

In 2006, NOAA contracted with an external team of evaluators to learn how Chesapeake B-WET 
programs implement MWEEs and what outcomes were achieved. They were specifically interested 
in knowing if professional development workshops increased teachers’ confidence in implement¬ 
ing MWEEs in the classroom and to what extent B-WET has improved students’ stewardship and 
academic achievement. The results were used by B-WET coordinators to document the effects of 
currently-funded programs, to inform future funding decisions, and to share findings with national 
education communities. 

Post-program questionnaires showed that teachers’ confidence in their ability to implement 
MWEEs increased as a result of their participation in professional development. Almost all of the 
teachers reported that they taught about the watershed or the Chesapeake Bay after participating 
in the professional development trainings, including the large majority of those who had not taught 
about the watershed or Bay in the past. In addition, most teachers who participated in MWEE 
professional development believed that their students were better prepared for their state’s stan¬ 
dardized tests as a result of adding MWEEs to their curriculum. Almost all teachers believed their 
students’ engagement in learning increased, a factor associated with student achievement. 

Pre/post tests showed that students improved in three of eight stewardship characteristics as 
a result of participating in B-WET-funded MWEEs. More importantly, students increased in the 
characteristic most closely associated with future behavior, their intention to act to protect the 
Chesapeake Bay watershed. Students’ knowledge of issues confronting the watershed or Bay 
and actions in which they can engage to protect the watershed or Bay increased. Students’ sense 
of responsibility to protect the environment and their feeling that they can make a difference 
appeared to be most positively influenced by collecting and analyzing data, conducting action 
projects, and reading about issues that impact the Bay. 

Source: Kraemer, A., Zint, M., and Kirwan, J. (2007). An Evaluation of NOAA Chesapeake B-WET and Training Program Meaningful 
Watershed Educational Experiences. Retrieved January 2009, from NOAA Chesapeake Bay Office website: http://noaa.chesa- 
peakebay.net/FormalEducation.aspx 
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Why is Evaluation Important to Project Design and Implementation? 

The reasons for conducting a project evaluation vary as dramatically as do the projects 
and their contexts. Perhaps the most common reason revolves around the desire to 
understand, in a systematic way, what is and is not working in a project. All too often, 
how projects work (or don't work) is understood primarily through a combination of 
instincts, anecdotes, and numbers on a balance sheet. The passionate instructor wrapped 
up in the moment of teaching is probably not in the position to make defensible 
claims about the long-term impact of a lesson on learner's education. Likewise, a series 
of anecdotes, although informative and even satisfying, cannot serve as the basis for 
generalizations. Evaluation provides perspective. It provides evidence. It provides the 
types of information necessary for sound decision-making. The following outlines 
additional benefits of conducting project evaluations. 

• Participants: Participants are core to the success of the project. In the long run, project 
sustainability will depend on the degree to which participants benefit directly, short¬ 
term and long-term, from the experiences or services. The evaluation will provide 
evidence of the ways in which participant learning is impacted. 

• Project Improvement: Project strengths and weaknesses can be identified through an 
evaluation. Of equal importance, an evaluation can map out the relationships among 
project components - that is, how the various parts of a project work together. This 
information can be used to re-design the project and increase both efficiency and 
effectiveness. 

• Public Relations: Data generated by an evaluation can be used to promote the products 
and services of the project within and outside of the agency. Instead of vague claims 
and uncertain assertions, statements based on evaluation results will be viewed as 
more substantial and justifiable. 

• Funding More and more education coordinators require a comprehensive, outcomes- 
based evaluation. They want to know what types of impacts the project has made. 

An evaluation can provide evidence of project effectiveness. Such evidence may be 
important when limited resources are being distributed internally. Evaluation results 
are often used in the process of determining if a project should be continued, scaled 
backed, discontinued, or enhanced. 

• Improved Delivery: Projects evolve over time. What was once a coherent, discrete set of 
activities may have grown into a jumbled set of loosely related events. An evaluation 
can help clarify the purposes of the project, allowing decision-makers to examine 
project components against well-thought out criteria. Valid comparisons of projects 
and activities can be made and duplication of efforts can be limited. It is quite 
possible that the evaluation will uncover a gem hidden in the jumble. 

• Capacity Building: Engaging staff members, volunteers, and other stakeholders in the 
design and implementation of an evaluation will provide opportunities for skill¬ 
building and learning. As the project or program is examined, those involved will also 
develop insights into the workings of the project and perhaps even the workings of the 
organization. These insights can be used to inform a strategic examination of projects 
and programs by identifying priorities, overlap, gaps, and model programs. 
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• Clarifying Project Theory; When the project was designed initially, it was built either 
explicitly or implicidy on a project theory that explained how things work or how 
people learn or even how organizations change. Based on experiences with the project 
and information taken from research literature, the evaluation provides an opportunity 
to revisit the theory behind the project. By making the project theory explicit, the 
underpinnings of the project and what makes it work will be better understood and thus, 
better implemented. Staff members and volunteers who understand why a particular set 
of teaching methods was selected or why the project activities were sequenced the way 
they were will be more likely to follow the plan. They will also feel more ownership in 
the project if they understand the theory behind the project more fully. 

• Taking Stock; Engaging in evaluation provides a precious opportunity to reflect on the 
project, to consider where the project is going and what it has been able to accomplish 
compared to expectations. Taking stock is more than accumulating information about 
the project - it is learning through the project. 


Steps in Planning a Project Evaluation 


PLANNING 

Step 1. Reexamine the issue, audience, and project objectives 

Step 2. Establish the planning team (including stakeholders, audience, and evaluators) 

Step 3. Identify a purpose for the evaluation 

Step 4. Focus on project improvement 

Step 5. Assess project outcomes and impacts 

Step 6. Clarify the time frame in which the activities and impacts (outcomes) are expected to occur 

Step 7. Perform a literature search 

Step 8. Select data collection methods and develop questions based on the evaluation goals and objectives 

DATA COLLECTION 

Step 9. Determine the audience sample 

Step 10. Design and pilot the data collection instrument 

Step 11. Gather and record data 

DATA ANALYSIS AND REPORTING 

Step 12. Perform data analysis 

Step 13. Manage data 

Step 14. Synthesize information and create a report 
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Planning an Evaluation 

As mentioned earlier, evaluation is the systematic collection of information about 
activities, characteristics, and outcomes of projects to make judgments about the project, 
improve effectiveness, and/or inform decisions about future programming. Consequently, 
great care must be made in the planning of any evaluation effort. Fourteen steps in 
conducting a project evaluation are outlined below. The outline is intended to breakdown 
a complex process into manageable steps. Please recognize, however, that in providing an 
overview of the process, nuances and detail are necessarily omitted. 

Planning 

Step 1. Reexamine the issue, audience, and project objectives 

Before a project evaluation can be designed, it is essential to fully understand the project 
- its components, the relationships among the components, the audience(s), and the 
intended outcomes (short-term, intermediate, and long-term). The project objectives (Part 
II, Step 3) and logic model (Part II, Step 4) should be reexamined and used as a road 
map for planning the evaluation. With the logic model and the associated performance 
objectives in hand, evaluation planners will be able to articulate how the project is 
supposed to work. From this foundation, the rest of the evaluation design will follow. 


Step 2. Establish the planning team (including stakeholders, audience, and evaluators) 

The project, in all likelihood, involves a variety of players. Project managers, resource 
managers, staff members, volunteers, participants, and community members all have a 
stake in the overall success of the project. Each plays a different role and sees the project 
through a different lens. These perspectives should be tapped when planning an evaluation. 
To ensure that ideas and perspectives are represented, members of stakeholder groups 
should be invited to participate in an evaluation planning team. The team, depending on 
the particulars of the evaluation, may play a purely advisory role or may take a more hands- 
on role in the actual data collection. The exact expectations of planning team members 
need to be decided on and articulated early in the process. Hopefully, this team was 
established during the planning process for the project itself (see Part II, Step 2). 


Step 3. Identify a purpose for the evaluation 

The evaluation team will need to determine the scope of the evaluation - that is, define 
the purpose of the evaluation, what is going to be evaluated, and who will use the 
evaluation. If the purpose is to assess the extent to which a program is operating as 
planned or to collect evidence of progress toward the outcomes identified in the TOP 
model then a process or implementation evaluation is called for. On the other hand, 
if the purpose of the evaluation is to judge whether the goals of the project have been 
reached then an outcomes evaluation (summative evaluation) is warranted. Given time 
and resource constraints, it may not be realistic to expect that all aspects of the project 
will be evaluated. The project team will need to set specific goals and objectives that can 
be used to focus evaluation planning and design. 
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What evaluation steps do people really do? 


This manual aims to provide you with a good picture of how programs should be planned, imple¬ 
mented, and evaluated. But when it comes to evaluation, what do people really do? 

Well, some really do everything! For example, the Multicultural Education for Resource Issues 
Threatening Oceans (MERITO) program of the Monterey Bay National Marine Sanctuary (MBNMS) 
conducted a needs assessment with representatives and community leaders to identify ways to 
expand marine conservation education and outreach efforts to local Hispanic communities. The 
project team at MBNMS then used logic modeling with stakeholders to develop, implement, and 
evaluate bilingual ocean and conservation-related outreach programs to Hispanic students, teach¬ 
ers, adults, and families living near the MBNMS. 

On the other hand, some programs do very little on a formal basis. But even in these programs, 
when asked how they know the program is successful, the educators are able to cite examples. 

Thus, data are being gathered, even if informally. 

Most programs fall somewhere in the middle. Education coordinators do pieces of several of the 
steps listed, but may not do all of them, or they do some of them cursorily. For example, Step 7, 
performing a literature review, works for both planning and evaluating the program. One educator 
knew she didn’t have the educational background to develop a program and so spent several weeks 
looking at what others had done with similar content programming. She felt that time spent on Step 
7 made a tremendous difference in the success of her program since she “was able to pick from the 
best of what’s out there already,” and she had clear insight into what she needed to measure in order 
to be sure her program was meeting its goals. Others hoped to avoid “re-inventing the wheel” and 
used listserves and correspondence with colleagues and friends to locate existing resources. A few 
educators also conducted Internet searches for resources, research, and reports, including visiting 
educational clearinghouses to locate well-reviewed materials and “best practices.” In some cases, 
they were able to find evaluation instruments that could be adapted to their own programs. 
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Step 4. Focus on project improvement 

The purpose of process or implementation evaluation is generally for project 
improvement. This type of evaluation focuses on what services are provided and to 
whom and how. The intent is to strengthen the program by providing feedback on its 
implementation, progress, and success. Useful information is collected early in the 
program so that changes can be made which will enhance program effectiveness, rather 
than waiting until the program is over. Therefore, process evaluation generally occurs at 
the lower four levels of the project evaluation staircase: resources, activities, participation, 
and reactions. 
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At its best, process evaluation is an essential decision making tool that can transform the 

project. Process evaluation can be used to: 

• Gather information/data about an audience's reaction to project activities or products/ 
materials. Changes may be made as a result of the positive or negative feedback to 
project activities or learning experiences. 

• Gather information/data about problems with project delivery and assess progress 
towards outcomes of a project during implementation. 

• Help provide information that can be used in making decisions about modifications, 
continuation, or explanation of the project. Results may be used to decide how to 
move forward with an existing project. 

Questions that might be addressed by process or implementation evaluation include: 

• Are the project resources (time, money, expertise) being utilized as planned? 
(Resources) 

• Is the project being implemented as planned? Are the intended activities, products, or 
services being provided? What promotional strategies worked or failed? (Activities) 

• Is the project reaching the target audience? Which targeted audiences participated 
in the activities? How extensively is the audience engaged in project activities? 
(Participation) 

• What are the participants' reactions to the project activities? Do participants perceive 
immediate benefits from their participation in the activities? (Reactions) 


Step 5. Assess project outcomes and impacts 

During project development, objectives or targets were set at some or all levels of the TOP 
programing staircase based on need or opportunity assessment. Outcome evaluation 
assesses the extent to which those project targets are reached. While outcome evaluation 
generally occurs at the upper three levels of the programming staircase (knowledge, 
attitudes, skills, and aspirations (KASA), behaviors or practices, and social, economic and 
environmental (SEE) conditions), it can focus on project outputs if the purpose of the 
evaluation is to understand how outcomes are produced. 

Each level of the TOP model provides different information about the outcomes of a 
project, from participant satisfaction to longer-term outcomes or impacts. Ascending 
TOP's project evaluation staircase provides stronger evidence of project accomplishments 
relative to the desired conditions. The project team should select levels of evaluation 
based on the type of information needed to evaluate the project accurately. 

Outcome evaluation informs decision-makers about the value or worth of the project. 
Results of an outcome evaluation provide the information necessary to make decisions 
about the continuation, revision, or expansion of the project. 
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Questions that might be addressed by an outcome evaluation include: 

• Did the participants increase awareness, knowledge, and or problem-solving skills? 
Did participants change their opinions, attitudes, or viewpoints as intended? (KASA) 

• Have participants changed their practices or behaviors as a result of the KASAs learned 
during their participation in the project? (Practice) 

• Have targeted social, economic, or environmental conditions improved as a result of 
KASA and practice changes made by the participants? (SEE conditions) 


Ascending TOP’S Project Performance Staircase 


Outcome 

evaluation 


Process or 
implementation 
evaluation 


Project Evaluation 


L 


SEE changes 


L 


Practices adopted 


L 


KASA gained 


L 


Participant reactions 


L 


Participants involved 


L 


Activities conducted 


L 


Resources used 


Process/Implementation evaluation. At startup, documenting the materials cost and staff time 
helps project coordinators explain the scope of the programming effort in terms of resources 
expended. Monitoring whether the targeted activities are implemented as planned and participa¬ 
tion levels are adequate provide evidence of program implementation. The education coordinator 
assesses whether the intended target audience is being reached and whether they are satisfied 
with the project as it was being implemented. 

Outcome evaluation. After the project is delivered, short-term outcomes focus on knowledge re¬ 
tained, attitudes changed, skills acquired, and aspirations changed. Intermediate outcomes at the 
practices level focus on the extent to which practices are implemented. Typically, these outcomes 
are measured a few months after the project. Intermediate outcomes lead to longer term social, 
economic, and environmental changes (impacts). 
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Step 6. Clarify the time frame in which the activities and impacts (outcomes) 
are expected to occur 

The relationships between overall project design, implementation, and project evaluation 
cannot be overestimated. Project evaluation is integral to project design. The evaluation 
timeline must be integrated into the project implementation timeline and vice versa. 
Otherwise, important opportunities, such as the ability to collect baseline data or assess 
critical impacts, can be easily missed and lost forever. Planning and implementing an 
evaluation can take several months. Sufficient care must be given to the development of 
the evaluation timeline to ensure effectiveness. 


Sample Evaluation Timeline for the 


Development and Analysis of a Survey 


Month 1 

Form evaluation team; set expectations; conduct background research (literature 
review, gather existing data, review logic model) 


Month 2 

Determine survey method; determine sampling method; develop draft survey 


Month 3 

Pilot test survey and revise 


Months 4-5 

Administer survey (mail, phone, in person) 


Month 6 

Code and enter data; analyze data 


Month 7 

L_ 

Write evaluation report and send results to stakeholders 



Step 7. Perform a literature search 

Evaluation, like overall project planning, rarely takes place in a vacuum. As mentioned 
previously, projects are developed based on explicit or implicit theories of how the world 
works. In designing an evaluation it is helpful to identify the related literature and use 
this literature as a touchstone. Likewise, research into evaluation processes, practices, 
and standards is useful. This is particularly true if the evaluation team does not include 
an outside evaluation expert or someone with significant evaluation expertise. Finally, 
existing sources of information (previous evaluations, census data, reports, budgets, etc.) 
should be tapped. 

Some useful sources of information and examples of project evaluations include: 

• NOAA, Coastal Services Center (http://www.csc.noaa.gov) 

• NOAA, National Marine Sanctuaries, Education Project Evaluation (http://sanctuaries. 
noaa.gov/education/evaluation) 

• Online Education Resource Library (OERL) (http://oerl.sri.com) 

• University of Michigan, My Environmental Education Evaluation Resource Assistant 
(MEERA) (http://meera.snre.umich.edu) 
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• University of Wisconsin-Extension, Program Development and Evaluation (http:// 
www.uwex.edu/ces/pdande) 

• Web Center for Social Research Methods (http://www.socialresearchmethods.net) 

Step 8. Select data collection methods and develop questions based on the 
evaluation goals and objectives 

By this point in the process, the evaluation team has determined why an evaluation is 
being conducted, who will conduct it, what will be evaluated, who will be evaluated, 
and when the evaluation will take place. Each of these decisions begins to define the 
type of evaluation (e.g„ process, outcome) to be conducted as well as the data collection 
instruments (e.g., surveys, interviews, case studies, focus groups) that will be most 
appropriate (see discussion of data collection instruments in Part IV). 


Data Collection 

Step 9. Determine the audience sample 

Except in rare cases when a project is very small and affects only a few participants, 
evaluations will be limited to a subset of the total anticipated audience. The preferred 
method for selecting a subset is random sampling - a procedure that reduces sample bias 
by selecting a sample that accurately reflects the population (see Appendix B). A sample 
represents the population if every person in the population has an equal chance of being 
selected. In general, to reduce sampling errors, make the sample as large as possible in 
terms of time and money. The larger the sample, the more it accurately reflects what 
would be obtained by evaluating everybody in the population. 

In determining an audience sample, some basic questions will need to be answered. 

For example: 

• How many audiences do you wish to sample? A project might identify teachers, parents, 
and students as its audiences. Each audience would sampled independently. 

• How many individuals do you hope to assess? See Determining Sample Size - Rules of 
Thumb in Appendix B as a guide. 

• How can you best reach the audience to collect data? Visitors to a marine sanctuary, for 
example, might only be accessed during a short window of opportunity; members of 
an advisory group might be more readily available. 

All of these questions are important in devising an appropriate evaluation sample. 
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Step 10. Design and pilot the data collection instrument 


Just as the initial design of the project required careful design and pilot testing of 
instructional materials to see how they worked, the data collection methods and 
instruments (e.g., interview, focus group, survey, observation) need to be crafted and pilot 
tested. The evaluator will need to consider a series of questions in order to design the data 
collection instrument, such as: How important is statistical precision? How important is 
in-depth information about one or more aspects of the project? How will data collection be 
standardized? Is contact information on target audience(s) available? What method of data 
collection would the audience be most receptive to? How many questions can be asked? 

There are many examples of instrument formats and styles. The project evaluators should 
make every effort to design an instrument that is appealing to the eye, simple to use, 
contains complete and explicit instructions, and collects the desired data. See Part IV for 
additional information on common data gathering instruments and Appendix D for 
examples of questions asked at each level of the TOP program evaluation staircase. 

Once the instruments have been designed, pilot tested, and revised, the instrument is 
ready to be used to gather data on project implementation and outcomes. 

Step 11. Gather and record data 

Again, just as the design of a project requires the consideration of various logistics (e.g., 
staff schedules, availability of facilities), the data collection process for an evaluation 
must be thoroughly scoped. The evaluation team will need to determine how data will be 
collected and recorded, and by whom. A system of coding and recording the data must be 
developed to ensure easy and accurate data analysis. Depending on the expertise of those 
responsible for data collection, it may be necessary to train interviewers, focus group 
facilitators, and/or observers. Additionally, evaluators will need to design a system that 
assures anonymity and that conforms to all aspects of ethical standards. 


Data Analysis and Reporting 

Step 12. Perform data analysis 

Analyzing quantitative and qualitative data is often the topic of advanced research and 

evaluation, ft is always a good idea to include a planning team member with survey 

and statistical analysis and/or qualitative data analysis expertise on the evaluation team. 

When that isn't possible, there are a few basics that can help in making sense of the data: 

• Have a plan in place for how to analyze, synthesize, store, and manage the data before 
starting the data collection. 

• Develop a plan to guarantee an unbiased analysis and reporting of the data. 

• Always start analyzing the collected data with a review of the evaluation goals and 
objectives. (Why was the evaluation undertaken? What question did the team want 
answered?) 
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• Make copies of all data, and store a master copy of the original data in a safe place. 

• For qualitative data, anticipate responses (the pilot test will help with this), and have a 
plan for categorizing and coding the data. 

Step 13. Manage data 

After the data are collected and even after the data have been analyzed, a plan must 
be put in place to continue the effective and ethical management of the data. In most 
cases, data remain viable for some period of time. After reading the evaluation report, 
decision-makers, other stakeholders, and other evaluators may generate questions that 
can be answered by revisiting the data. Consequently, it is important to develop a plan 
for continued access to uncorrupted data. If data are to be retained for some period of 
time, the project team must also make certain that the confidentiality and anonymity of 
respondents is maintained. Finally, intellectual property rights - including the question 
of who "owns" the data - need to be defined. 

Step 14. Synthesize information and create a report 

After the data have been collected and analyzed, an evaluation report must be written. 
Obviously, knowing when the evaluation report and its recommendations will be needed 
impacts the overall timeline as well as decisions related to evaluation design and data 
collection strategies. Although the nature of the evaluation report (e.g., length, style 
of writing) is determined to some extent by its intended audience, there are standard 
components to any evaluation report. For example, the report must include a description 
of the evaluation methods used: Flow were the samples drawn? How was data collected 
and analyzed? In addition, the report must include a discussion of the problems 
encountered and errors made with the design and implementation of data collection 
instruments or tools. Readers of the report must feel confident that the results of the 
evaluation are credible and the recommendations sound. 

Who Should Conduct the Evaluation? 

Early on in the evaluation planning process, the decision will need to be made whether 
or not to hire an outside evaluator. In some cases, the decision may have been made for 
the team. Many managers require that an outside evaluator be contracted. An outside 
evaluator is seen as an objective third-party who may be able to bring a fresh perspective 
to the project. In addition, a professional evaluator has expertise that may not exist 
in-house. Many funding agencies require external evaluators to conduct evaluations 
to minimize bias and provide credible evidence of project achievements. If evaluation 
expertise does exist and an outside evaluator is not required, it is possible to ran major 
components of the evaluation in-house. However, managers should proceed with 
caution. Those with strong ties to the project (i.e., project manager, staff members, 
volunteers, advisory committee members) may find it difficult to shed their biases, 
particularly if evaluation results are to be used in decision-making. Project staff members, 
volunteers, and other stakeholders should be involved in determining the focus and 
objectives of the evaluation. However, at a minimum, an outside evaluator should be 
responsible for data analysis and interpretation. 
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Does evaluation really matter? 


“We all know evaluation is important.. .some of what we learned made me realize just how im¬ 
portant.” In interviews with a variety of agency educators, a common theme was that evaluation is 
important, but that it’s often not thought about until after a program is well underway. 

Most educators used feedback forms for the bulk of their evaluation information. In most cases, 
they found that people liked their programs and offered some minor ways in which the program 
could be improved. Several of the educators who used evaluation in this way commented that 
there was no need for more information because they know their program already. Many wanted 
to continue with gathering the same type of evaluation data in order to maintain a consistent level 
of participant/audience satisfaction with the program. 

On the other hand, educators who wanted to use evaluation to improve their programs asked more 
people more questions in different ways and at different times in order to get a “good picture of 
how we’re doing.” The outcomes from these evaluations revealed that people liked the programs, 
but also gave the educators a lot of information for making changes and for reporting purposes. 

One educator was surprised when she realized that not all the findings had to be high scores 
because she was able to explain how the data were being used and able to show improvement. 

“That made my boss very happy -1 was able to show accountability.” 

When evaluative data are gathered reveals a lot about program managers’ beliefs about evalua¬ 
tion. A clear majority assume that a feedback form is “an evaluation,” while others believe it is a 
pre-test/post-test. Some individuals used team meetings in the planning phase all the way through 
staff and secondary audience measures in the outcome measurement phase as “parts of how 
we’re learning about our program.” 

As with needs assessment, the larger the program, the more time that it takes to conduct and run, 
and the more funding it has, the more evaluation is formalized throughout the process. This makes 
good sense from an accountability perspective - where you spend your resources, including your 
time, makes all the difference, and evaluation can help you improve and understand whether your 
resources are being wisely used. 

V_ J 


Evaluation Costs 

Depending on the size and complexity of the project, it is typically recommended that 
5 to 15% of project costs be set aside for evaluation. Costs may be lower if some or even 
most of the evaluation is conducted in-house. (Remember, that even though a check may 
not be written to an outside evaluator, staff time still involves costs.) It should also be 
remembered that a great deal of data may already exist - for example, in annual reports 
or participant exit interviews. Obviously, building an evaluation around existing data has 
its pitfalls, but a critical examination of quality sources of existing information that can 
help answer the evaluation questions should not be overlooked. 


Part III. Project Evaluation 
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Guidelines for Conducting a Successful Evaluation 

1. Invest heavily in planning. 

2. Integrate the evaluation into ongoing activities of the program. 

3. Participate in the evaluation and show program staff that you think it is important. 

4. Involve as many of the program staff as much as possible and as early as possible. 

5. Be realistic about the burden on you and your staff. 

6. Be aware of the ethical and cultural issues in an evaluation. 

From: U.S. Department of Health and Human Services. The Program manager’s guide to evaluation. 


Ethics 

Virtually all evaluations involve collecting some information, directly or indirectly, 
from individuals. In designing the evaluation, the project team must ensure that the 
individuals involved are treated with respect and sensitivity. Not only should the 
maintenance of confidentiality and/or anonymity be a high priority, but the time and 
effort expended by evaluation participants should be respected. It is not appropriate, 
for example, to collect data from individuals when there is no specific plan to use that 
data. Respondents' physical and psychological well-being must be assured throughout 
the data collection process. Interviews, surveys, etc. must be designed in such a way that 
evaluation participants are not embarrassed or asked to do something that might put 
them in jeopardy. Whether they are providing demographic data on a survey, completing 
a test, or responding to an interview, respondents are disclosing personal information 
about themselves. Respectful data collection is not enough, however. Evaluation ethics 
require that respondents understand that they are participating in an evaluation and give 
their permission (with minors, a parent or guardian must provide informed consent). 


Part III Wrap-Up 

Evaluation is not a frill or luxury for education programs to be conducted if there are 
sufficient funds remaining. Rather, an evaluation should be built into the plans for a project 
from the start. A well-crafted project evaluation helps decision-makers make decisions by 
determining if the project works and whether or not it is worth the investment of time and 
resources. Evaluation results help justify worthwhile projects in future funding cycles and 
help discontinue projects that have outlived their usefulness. Equally important, the data 
from a well-conducted evaluation can also help shape future efforts so as to provide more 
effective, better targeted, and more widely used programs to the public. 

The next section of this manual provides an overview of some of the most commonly- 
used data collection instruments. 
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Part IV. 

Data Collection Instruments 



Introduction 

Project coordinators and evaluators have an array of data collection instruments available 
to them. Although selecting the most appropriate instrument requires thought and 
careful consideration, the selection process is also shaped by a number of decisions that 
have already been made. The type of evaluation being considered (i.e., needs assessment, 
process, or outcome) will determine, to some extent, the most appropriate data collection 
instmment(s). Likewise, the level of evaluation being conducted (i.e., KASA changes, 
practices, SEE conditions), the audience involved (e.g., children vs. adults, casual visitors 
vs. organized groups), and the amount of resources available (e.g., time, money) will all 
help determine which data collection instruments should be used. Each data collection 
strategy comes with strengths and weaknesses. 


Matching Data Collection Instruments to What is Being Assessed 

It is often the case that an evaluation would want to address the degree to which 
participants increased their level of understanding, developed a particular set of skills, 
or further considered their attitudes on a topic. Some data collection instruments are 
particularly adept at assessing knowledge gain. Others are appropriate for documenting 
skill or attitude development. The following chart provides some ideas of the 
appropriateness of using specific kinds of data collection instruments to assess knowledge, 
skills, attitudes, and behavior. 


Data Collection Instruments 

Knowledge 

Skills 

Attitude 

Behavior 

Interview 

X 


X 

(X) 

Focus group 

(X) 


X 


Questionnaire and survey 

X 

X 

X 

(X) 

Observation 


X 


X 

Literature review* 

X 

X 

X 

X 

Test 

X 

X 



Concept maps 

X 


(X) 


Document or product review 

X 

X 

(X) 


Case study 

X 

X 

X 

X 

Notes: 





(X) Indicates that this technique may be, but is not always, appropriate to evaluate the indicated type of learning. 


* For comparison from past to initial condition. 





Source: American Society for Training and Development (1989). Evaluation tool use. Alexandria, VA: Author. 
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Validity and Reliability of Evaluation Instruments 

Before you begin to collect data for your project evaluation, make sure the instruments 
or tools you plan to use have been validated. Anytime you develop instruments to collect 
data, whether it is for a needs assessment, a process, or an outcome evaluation, you 
should take steps to ensure the instruments are valid and reliable. Two factors that should 
be taken into consideration when validating your evaluation instruments are validity 
and reliability. The validity of an instrument is the extent to which it measures what it 
purports to measure. Test the validity of your evaluation instruments by asking a panel 
of experts if your questions adequately sample the content you wish to convey (content 
validity). The reliability of an instrument is the extent to which it yields consistent 
responses each time it is administered. 

One of the easiest ways to measure the validity and reliability of your evaluation 
instruments is through field or pilot testing. Field or pilot testing means trying out your 
instruments under conditions similar to those you expect to have for the evaluation, 
therefore conduct the field test on a representative sample of people who are likely to 
participate in the actual project. Field tests answer such questions as: Is the questionnaire 
providing the information needed to answer my evaluation questions? Are multiple 
interviewers collecting information in the same way? How consistent is the information 
obtained with the questionnaire? How accurate is the information obtained with the 
observation rubric? 

While these are probably the simplest ways to validate your instruments, there are a 
number of other more informative techniques you can use to establish reliability and 
validity. However, these require use of statistical methods. In many cases it is best to 
work with someone with expertise in statistical analysis for education projects when 
determining validity and reliability of your instruments. 

Obviously, care must be taken to ensure that your data gathering instruments are 
both valid and reliable. The approach you choose will depend on the expertise of the 
evaluation team, data collection methods, longevity of the project, and practicality in 
terms of time and resources expended. 


Mixed Methods 

Mixed methods, in evaluation, refers to the practice of using some combination of 
both quantitative and qualitative data gathering. Quantitative methods allow us to 
count events or number of participants, determine cost/participants, perform statistical 
analyses (mean, median, mode, standard deviation), and complete other calculations. 
Quantitative methods allow us to generalize the findings beyond the actual respondents 
to the relevant population. 

Qualitative methods allow us to record explanations, perceptions, and descriptions of 
experiences - often in the participants' own words. Qualitative methods allow us to 
create narratives that provide an in-depth view and a more complete understanding of 
the context of the evaluation. Typically, a small number of individuals participate in a 
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qualitative evaluation. Consequently, the results of this small number of participants 
cannot be generalized to the population. 

Each instrument has its own strengths and weaknesses. Using quantitative methods or 
qualitative methods in isolation limits what can be learned from the evaluation, what 
can be reported, and what can be recommended, with any confidence, as a result of 
the evaluation. Used in combination, however, the individual strengths of quantitative 
and qualitative methods can be maximized and the weaknesses minimized. More 
importantly, a synergy can be generated when using mixed methods. Results from more 
than one method of data collection can be "triangulated," providing greater validity 
and enhanced understanding. A survey of participants may provide a great deal of 
information about what services are most desired (and least desired); an interview of 
a small number of the participants may provide in-depth information concerning why 
those services are most desired (or least desired) and, importantly, what characteristics 
make a particular type of service most desired. 


Types of Data Collection Instruments 

The following table summarizes the purpose, advantages, and challenges of using 
different data collection instruments. Remember that since data gathering instruments 
are developed for a specific purpose and project, they rarely represent a pure form. For 
example, a survey or interview may include test items. A case study often incorporates 
observation, document review, and in-depth interviews. Appendix A contains detailed 
fact sheets for each of the nine data collection instruments summarized here. 
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Uses, Benefits, and Limitations of Various Data Collection Instruments 


Instruments 

Overall Purpose 

Advantages 

Challenges 

Interviews 

To fully understand someone’s 

• Provides full range and depth of 

• Can take much time 


impressions or experiences, or 

information 

• Can be hard to analyze 


learn more about their answers to 

• Develops relationship with 

and compare 


questionnaires. 

respondent 

• Can be costly 



• Allows for follow-up questions 

• Interviewers can bias responses 

• Generalization is limited 

Focus Groups 

To explore a topic in depth 

• Quickly and reliably obtain 

• Can be hard to analyze 


through group discussion, e.g., 

common impressions 

responses 


about reactions to an experience 

• Can be efficient way to gather 

• Need a good moderator to steer 


or suggestion, understanding 

range and depth of information 

the discussion and provide 


common complaints. Useful in 

in short time 

closure 


evaluation and marketing. 

• Can convey key information 

• Difficult to schedule 6-8 people 



about projects 

together 

■ Strong individuals may influence 
others’ responses 

Questionnaires 

To quickly and/or easily obtain a 

• Can complete anonymously 

• Might not get careful feedback 

and surveys 

lot of information from people in a 

• Inexpensive to administer 

1 Wording can bias client’s 


non-threatening way. 

• Easy to compare and analyze 

responses 



• Can administer to many people 

• Impersonal 



• Can obtain lots of data 

• In surveys, may need sampling 



• Many sample questionnaires 

and statistical expertise 



already exist 

• Doesn’t yield full story 

Observation 

To gather accurate information 

• Can view operations of a project 

• Can be difficult to interpret 


about how a project actually 

as they are actually occurring 

behaviors 


operates, particularly about 

• Can adapt to events as 

• Observations can be difficult to 


processes. 

they occur 

categorize 

• Can influence participant's 
behaviors 

• Can be expensive 

Literature Review 

To gather information on the 

• Can provide much information in 

• Can be out-of-date (e.g., 


audience and/or the issue. Identify 

relatively little time 

technology needs) 


what previous investigations have 

• Makes use of already gathered 

• Data synthesis can be difficult 


found about the knowledge, skills, 

information 

• May not address specific 


behaviors, or attitudes of the 

• Helps to sort changes over time 

questions of concern 


intended audience with relation to 

• Provides evidence about 

• Not a flexible means to get data; 


the issue. 

the problem 

data restricted to what already 



• Minimum effort or interruption of 

exists 



audience 

■ Statistical data may not address 
perceptions of the problem, or 
may not address causes of the 
problem 

• Reports may be incomplete 
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Uses, Benefits, and Limitations of Various Data Collection Instruments (cont.) 


Instruments 

Overall Purpose 

Advantages 

Challenges 

Tests 

To determine the audience’s 
current state of knowledge or skill 
regarding the issue. 

• Helps identify a problem or a 
deficiency in knowledge or skills 

• Results are easily quantified 

• Individual performances can be 
easily compared 

• Limited availability of validated 
test for specific situations 

• Language or vocabulary can be 
an issue 

• People may be concerned about 
how results will be used 

• Adults may resent taking tests 

Concept Maps 

To gather information about 
someone’s understanding of, and 
attitudes towards, a complex 
subject or topic. 

• Can offer a more comprehensive 
and complex view of someone's 
thinking than a test does 

• Could be a better tool for visual 
learners or test-phobic people 

• Produces qualitative and 
quantitative data 

• Takes training to complete 
properly 

• Takes training to administer 

• Can be challenging and time 
consuming to score 

• Can be difficult to analyze and 
interpret 

Document or 
Product Review 

To gather information on how 
the project operates without 
interrupting the project; comes 
from review of applications, 
finances, memos, minutes, etc. 

• Yields historical information 

• Doesn’t interrupt project or 
client’s routine in project 

• Information already exists 

• Few biases about information 

• Often takes much time 

• Information may be incomplete 

• Need to be quite clear about 
what one is looking for 

• Not a flexible means to obtain 
data; data restricted to what 
already exists 

Case Studies or 
Peer Review 

To fully understand or depict client’s 
experiences in a project; to conduct 
comprehensive examination 
through cross comparison of 
cases. 

• Fully depicts client’s experience 
in project input, process, and 
results 

• Powerful means to portray 
project to outsiders 

• Usually quite time consuming to 
collect, organize, and describe 

• Represents depth of infor¬ 
mation, rather than breadth 

• Information gathered represents 
a single individual or event; 
cannot be generalized 


Source: McNamara, C, (1997-2008). Basic guide to program evaluation. Retrieved December 2003, from Free Management Library website: 
http://managementhelp.org/evaluatn/fnLeval.htm 
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Selecting the Right Data Collection Instrument 

The tables below provide a convenient reference for the selection of appropriate evaluation data collection instruments for different 
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Literature review and document review utilize existing sources of information that are not dependent on audience type. 





B. Instrument versus Activity/Project 


o o. 
o cz 


o 

o 


o 

o 

Q_ 


O 

O 

Q_ 


TD 

O 

O 

CD 


O 

O 

CL 


td 

O 

o 

CD 


TD 

o 

o 

CD 


o 

03 


03 

C/3 


O 

o 

Q_ 


O 

O 

Q_ 


*o 

o 

o 

CD 


o 

o 

Q_ 


cd 

O 

<d 

O 

i— 

c_> 

O 

i— 

O 

< 

TD 

O 

TD 

O 

O 

o 

CD 

O 

o 


O 

o 

CD 

CD 

' -Lm 

CD 

CL 


CD 

CD 


(/) 

,03 


03 

LL 

O 

TD 

o 

TD 

o 

i— 

i— 

i— 

o 



i— 

i— 

i— 


o 


i— 

H— 1 

o 

o 

CD 

'os 

o 


03 

03 

03 

CD 

CD 

o 

03 

'os 

TD 

O 

CD 

CD 

LL 

LL 

CL 


LL 

LL 

LL 

LL 

LL 

Q_ 

LL 

LL 

O 

CD 
















03 

> 

3 

C/D 


o 

o 

CD 


TD 

o 

o 

CD 


O 

o 

CL 


CD 

O 


<D 

O 

TD 

O 

TD 

O 

O 

[2 

o 

O 

o 

CD 


CD 

CD 

CD 


O 

O 

CD 


TD 

O 

O 

CD 


TD 

O 

o 

CD 


o O 
O j- 
LL. CD 


o 

o 

CL 


*o 

o 

o 

CD 


■a 

o 

o 

CD 


TD 

O 

o 

CD 


"O 

o 

o 

CD 


£ 

03 


O 

o 

CL 


CD 

o 

TD 

O 

TD 

O 

i— 

TD 

O 

TD 

O 

TD 

O 

CD 

o 

TD 

O 

TD 

O 

TD 

O 

TD 

O 

TD 

O 

o 

o 

o 

03 

o 

o 

O 

O 

o 

o 

o 

O 

o 

CD 

CD 

CD 


CD 

CD 

CD 

CD 

CD 

CD 

CD 

CD 

CD 


CD 

O) (— 


O 03 
4= C 

s i 

£S to 


o 

.8 

I 


■E 

o 


_03 

,03 


CD 

> 

CD 

_03 

O) 


_Q3 

CL 


o 

E, 

'— 

O 

-T— 

-E 

C/3 

C/3 

03 

_E 

TD 

CD 

1 

CO, 

DsC 

03 

C= 



O 

5 

CD 

CO 

03 

O 

|2 

o 

|2 


CD 

S 


CD 

Q. 


03 

E 


o 

03 

CL 


E :-E 


CO 

C 

O 


.o 03 .CJ .ro 

b 'co S ^ 

=5 .^3 =5 ^ 

O ^ o_ ^ 


Q 

O 

o> 

3 , 

cd 

TD 


Part IV. Data Collection Instruments 


63 


'Literature review and document review utilize existing sources of information that are not dependent on the type of activity. Case study involves creating a detailed description of a project or a participant’s experiences 
with a project. Data for a case study can be collected through a combination of methods. The appropriateness depends, consequently, on the combination of methods used. 







Appendices 



Appendix A. 

Fact Sheets on Data Collection Instruments 



The appendix contains individual fact sheets that describe in more detail the nine 
data collection instruments - interviews, focus groups, questionnaires and surveys, 
observation, literature review, tests, concept maps, document or product reviews, and case 
studies. For each method, the number of participants, time and cost factors, and benefits 
and limitations are described. 

Interview 

What is it? 

Active interchanges between two people either face to face or via technology (e.g., 
telephone, email). 

How many respondents/participants? 

Typically 5-20. The number of individuals interviewed can be increased, but time and 
cost also increase. 

Time issues? 

Interviews can last 15 minutes to 2 hours, depending on depth. Data analysis is often 
slow and time consuming. (Data analysis can take weeks, especially if long interviews are 
transcribed and initial analyses are returned to respondents for a check of accuracy.) 

Cost issues? 

Project managers must factor in the cost of hiring/training interviewers, transportation 
costs (if interviewers must travel to meet those being interviewed), and substantial data 
analysis time. 

When to use it? 

Interviews are best used when in-depth information or a variety of perspectives about a 
topic, experience, or service are desired. Often, interviews are selected when the issue(s) 
is complex. Since broad, open-ended questions can be asked, interviews are appropriate 
when project evaluators do not feel that they can adequately anticipate types of 
responses. Interviews should also be used when literacy is an issue. 
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What are some of the benefits? 

• Variety of perspectives can be elicited 

• Can be very useful way to build rapport with audience/participants 

• Can generate broad and deep data about system or product 

• Interviewer can clarify questions and ask for clarification of responses (follow-up 
questions) 

• Interviewer can receive additional information in the form of nonverbal clues 

• Questions can be adapted if difficulties arise 

• Open-ended questions and a reduced amount of structure allow for new (unplanned 
for) information to be gathered 

• Interviewer can ask for more information than people would want to write in a survey 

• Respondents use their own words 

What are some of the limitations? 

• Bias due to data collector's interest and interpretations 

• Time intensive 

• Self-reporting of participants may bias data 

• Discussion can wander from purpose of interview — results may not be focused 

• Unskilled interviewers can make respondents feel self-conscious 

• Unskilled interviewers may gather poor data 

• Variations occur if there's more than one interviewer 

• Open-ended responses can be difficult to organize and analyze 

• Difficult to capture everything said unless taping the interview 

• Small sample 

• Replication is difficult 
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Focus Group 

What is it? 

A structured, interactive exchange between an interviewer/facilitator and a small group 
of people. 

How many respondents/participants? 

6-10 participants per focus group. More than one focus group is often helpful, 
especially if there is more than one audience involved in the project (e.g., parents, 
teachers, administrators). 

Time issues? 

Focus groups typically last about 1 to P /2 hours. Data analysis requires transcribing 
the focus group discussion and pulling trends from it. This process is relatively quick 
if only one or two focus groups has been conducted. 

Cost issues? 

Relatively inexpensive unless a focus group moderator is hired. Other cost considerations 
include: transportation for participants, room rental, food or other refreshments for 
participants, honorarium for participants. 


When to use it? 

Focus groups, like interviews, are best used when a variety of perspectives about a topic, 
experience, or service are desired. Because focus groups involve several individuals, they 
are particularly useful when there is reason to believe that peer pressure and/or the social 
nature of the situation will stimulate creativity and/or encourage discussion of conflicting 
points of view. Focus groups are best used when the topics are narrow or individuals have a 
limited amount of information about the topic to share - that is, the discussion is focused. 
A rule of thumb is that focus groups are best used when any one participant could only talk 
about the topic for ten minutes. 


What are some of the benefits? 

• Input can come from wide range of people and perspectives 

• Participation may have positive public relations impacts 

• Can clarify different points of view 

• Can provide a good indication of the root of a problem 


What are some of the limitations? 

• May represent special interests 
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• Participants may use as "gripe session" 

• May heighten expectations beyond what can be realistically provided 

• One participant may influence attitudes and opinions of others 

• Need to transcribe and code information for analysis; hard to quantify 

• Cannot capture all information without taping session; not all people are comfortable 
being taped 

• Small sample size 
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Questionnaire and Survey 

What is it? 

Data collection instrument through which individuals respond to printed or oral 
questions. May be recorded by either respondents or data collector. 

How many respondents/participants? 

25-1000 (typically). The number of surveys is limited primarily by time and money. 

Some instruments (e.g., email surveys) can be sent to any and all members of the 
audience who have access to email. 

Time issues? 

20-45 minutes to complete; two to three months to collect data (for a typical mail 
survey; much shorter for phone, group or email surveys); one or more months to analyze 
data (assuming most questions are closed-ended). 

Cost issues? 

Costs include printing, postage, return postage, follow-up postcards. (Costs increase with 
a large sample size.) If surveys are administered in person (either individually or in small 
groups), costs can diminish dramatically. Email surveys are least costly, but the number of 
questions that can be asked is limited. 


When to use it? 

Surveys allow for the systematic and standardized collection of data that can be 
generalized to the population (assuming proper sampling procedures were followed). 
Surveys are appropriate when self-reported data about knowledge, attitudes, skills, 
and behaviors are desired. Because of their format, surveys can be administered to a 
large number of people individually (e.g., in person, email, mail) or in groups (e.g., 
participants in a workshop). In addition, surveys are particularly useful when potential 
respondents are dispersed geographically. When evaluators have a good idea of the types 
of responses expected, surveys offer an efficient method of collecting information. 


What are some of the benefits? 

• May be easiest to quantify, summarize, and report on the data 

• Time-effective for use with geographically dispersed or large sample (respondents 
complete and return) 

• Large sample size; data can be generalized to population 

• Range from inexpensive to expensive (depending on design and administration) 
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• Can provide opportunity for expression without fear of embarrassment (anonymity) 

• Can (should) be designed to be relatively bias-free 

• Questions from other instruments can be used or modified 

• Can gather qualitative and quantitative data 

• Respondents can complete at their convenience (for written instruments) 

• Useful at all evaluation stages 

• Good for gathering information that requires sequencing (respondents can't read 
ahead if given as an oral survey) 

• Survey administrators can clarify questions if conducted in person or over the phone 

• Easily adaptable to a wide variety of environments 

What are some of the limitations? 

• May have limited provision for unanticipated responses 

• Not adaptable once the survey is distributed 

• Requires significant time and high level of expertise to develop valid surveys 

• Low return rates for some survey formats (e.g., phone, mail) can skew data 

• Can be impersonal (written, self-response format) 

• Questions may miss true issues 

• Questions and answers can be interpreted differently 

• People have been negatively conditioned to the value of surveys 

• Language or vocabulary may be an issue 

• People tend to want to get the "right" answers (even if the questions is asking for 
attitudes) 

• The survey administrator can influence the respondents 

• People may hurry through answers without thinking about them 
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Observation 

What is it? 


Data collection based on watching a process or skill and systematically recording events. 
Observations may be made in person (at the time of the event) or using media (e.g., 
analysis of a video tape recording). 


How many respondents/participants? 

Typically 5-20. The number of people or objects/events observed depends on the subject 
being observed (e.g., short, discrete activity such as use of a recycle container vs. complex, 
lengthy activity such as the ability to facilitate a lesson). 


Time issues? 

Varies depending on what is being observed. Counting the number of cans recycled 
during a single event (e.g., community festival) requires little time commitment. 
Observing group interactions over the life of a project requires more complex data 
collection (and therefore analysis), and a longer time commitment. 


Cost issues? 

Varies with the complexity of the subject. 


When to use it? 

Observation allows evaluators to document behavior. When evaluators want to know 
how people behave (e.g., demonstration of skills, recycling, fishing) or the results of 
specific behavior (e.g., diversity of plants or animals in a restoration area), observation 
should be used. Actual behavior (or the results of the behavior) is documented, not what 
people say they do or are planning on doing. 


What are some of the benefits? 

• Little interruption of work flow or group activity (if done properly) 

• Generates data about actual behavior, not reported behavior 

• Can see project in action 

• Can provide good in-depth data 

• Data collected in context 

• An astute observer can recognize interaction problems not easily described by 
participants 

• Observer can follow action at different points in the system 

• Administrative costs can be kept to a minimum 
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What are some of the limitations? 

• Requires process and content knowledge by observer 

• Observer can disrupt or alter the system 

• Observer can be considered a spy 

• Data can be skewed by observer's biases (and skills) 

• Data are not always quantifiable and may require judgments by the observer 

• Typically, small sample size 

• Usually time intensive 

• Does not indicate how participants view their actions 

• Replication difficult 


Using Rubrics for Scoring Performance 


A rubric is a set of criteria and standards linked to learning objectives used to assess a perfor¬ 
mance. When we consider how well a project participant performed a task we place the perfor¬ 
mance on a continuum from “exceptional” to “not up to expectations.” In addition to using rubrics 
when observing a performance, rubrics also provide a template for scoring portfolios, open-ended 
questions, and group or independent research projects or reports. With a well written rubric, it is 
reasonable to expect that all performances will be measured with the same yardstick. Additionally, 
when rubrics are used in educational contexts and provided to learners prior to completing the as¬ 
signment, learners know what is expected of them and they are likely to perform better. 

Rubrics are comprised of three components: criteria, descriptors, and levels of performance. For 
each criterion, the evaluator assigns different levels of performance. Descriptors spell out what is 
expected of learners at each level of performance for each criterion. A descriptor tells the learner 
precisely what performance is required at each level and how their work may be distinguished from 
the work of others for each criterion. Similarly, the descriptors help the instructor more precisely and 
consistently evaluate the learners’ work. 
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Sample Rubric - Taking Water Samples 


Score 

Safety: Degree to 
which learner 
follows correct 
safety procedures 

Procedures: Degree to 
which learner follows 
proper mechanics in 
water quality analysis 

Results: Degree 
to which learner 
obtains proper 
sample values 

Interpretation: 

Degree to which 
learner develops likely 
hypotheses 

o 

Fully meets 
standards 

Handles chemicals and 
glassware safely 

Obtains uncontam¬ 
inated samples and 
follows correct steps 
for pH analysis 

Both samples within 

0.3 points of the 
correct pH 

Can list three plausible 
reasons why the pH 
of the two samples 
differs and can defend 
reasoning behind 
hypotheses 

e 

Partially meets 
standards 

No serious safety 
issues during analysis, 
but procedures 
deviate from ideal 

Has some problems 
following instructions, 
but procedure 
adequate for 
approximate correct 
test results 

One sample within 

0.3 points of the 
correct pH 

Can list two plausible 
reasons why the pH 
of the two samples 
differs and can defend 
reasoning behind 
hypotheses 

Q 

Major departure 
from some aspect 
of standards 

Shows some concern 
or knowledge about 
safety issues, but 
is careless in 
handling materials 

Major problems 
with procedures 
that will likely yield 
incorrect results 

Neither sample 
within 0.3 points, but 
at least one sample 
within 0.5 points 

Can list one plausible 
reason why the pH 
of the two samples 
differs and can 
defend reasoning 
behind hypothesis 

o 

Does not meet 
standards 

Disregards safety 
concerns when 
handling materials 

Does not follow 
necessary steps in 
analysis and cannot 
obtain useful results 

Neither sample within 
0.5 points 

Cannot list even one 
plausible reason why 
the two samples differ 
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Literature Review 

What is it? 


Existing information in the form of reports, historical data, planning and budget reports, 
organizational structure charts, workshop evaluations, career development reports. Also 
includes published research and other evaluation reports. 

How many respondents/participants? 

N/A 

Time issues? 

Varies - depends on the number of documents, their availability, and the amount of 
information being analyzed. 

Cost issues? 

Relatively inexpensive, using existing documents and data. 

When to use it? 

Literature reviews are used primarily in early stages of the development of an evaluation 
and are particularly useful in front-end evaluation. Existing literature (e.g., theory, 
research findings, previous evaluation reports) and data (e.g., test scores) can provide a 
baseline. Other forms of existing data (e.g., budget reports, workshop evaluations) can 
help paint a picture of the intended audience, their perceptions and their reactions. 

What are some of the benefits? 

• Can be less time consuming 

• Makes use of already gathered statistical data 

• Easier to chart changes over time 

• Provides excellent evidence of problem (needs assessment) 

• Minimum effort or interruption of workers 

What are some of the limitations? 

• Data synthesis can be difficult 

• May not address specific questions 

• Data on the causes of problems may not have been collected 

• Reports may be incomplete 

• Organizations can be hesitant to share if results reflect poorly on the organization or 
a project 

• Reports may have been adjusted or "selectively edited" 
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Test 

What is it? 


An exam that assesses knowledge or skill level. Can be essay, fill in the blank, true/false, 
and/or multiple choice formats. 

How many respondents/participants? 

From 25 to thousands. Numbers are limited by the length of the test (think SAT or GRE) 
and its format. Scantron (or bubble tests) can be administered to hundreds or thousands 
(assuming access). Essay tests are limited by the ability to score them. 

Time issues? 

10 minutes-1 hour to administer (although most participants would resist a test that 
takes more than a few minutes to complete). Development of a valid and reliable test 
instrument can take months. Data analysis, if closed-ended questions are used, should 
take a short amount of time. 

Cost issues? 

Inexpensive to moderately expensive. Greatest cost issues revolve around the 
development of the test (ensuring validity and reliability). If instruments exist or simple 
measures of knowledge/skills are used, development costs will be limited. 

When to use it? 

Tests are used when evaluators want to assess the audience's level of knowledge or 
skills. Tests measure a point in time; they cannot predict future or past performance. 

If administered at intervals, tests can provide an indication of change (e.g., increased 
understanding) over time (pre/post assessments; longitudinal). 

What are some of the benefits? 

• Helps identify level of knowledge or skill (achievement and accountability) 

• Results are easily quantified 

• Individual performances can be easily compared 

• Helps determine if intervention has made a difference in knowledge or skill level 

What are some of the limitations? 

• Limited availability of validated tests for specific situations 

• Validity issues - does it test the appropriate knowledge and skills 

• Language or vocabulary can be an issue 

• People can be very concerned with how test results will be utilized (especially adults) 

• Adults sometimes resent taking tests, which typically have a negative connotation 
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Concept Map 

What is it? 

A graphic representation of how an individual thinks about a topic. Concept maps depict 
how the mind is able to organize information in a hierarchical manner and how new 
information is connected to existing information. 

How many respondents/participants? 

Typically 5-20. The number of individuals asked to produce a concept map can be 
increased, but time and cost also increase. 

Time issues? 

5-45 minutes. Before data can be collected, respondents will need to learn how to draw 
a concept map. Data analysis is slow and time consuming (can take weeks, especially 
if concept maps are complex - showing many relationships among ideas - or if a large 
number of individuals produced concepts maps). 

Cost issues? 

Analysis of concept maps requires expertise and training. 

When to use it? 

Concept map is best used when the evaluator wants to understand the relationships 
among ideas or attitudes. Because it is structured differently from a test or typical survey, 
constructing a concept map may be less intimidating for some (especially for those who are 
uncomfortable with tests). 

What are some of the benefits? 

• Provides a visual representation of how individuals see the topic and the relationships 
among ideas 

• Tends to reduce test anxiety (people often see it as a fun exercise rather than a test 
of understanding) 

• Allows respondents to illustrate complex ideas and relationships (particularly useful 
for visual learners) 

What are some of the limitations? 

• Difficult to score (data analysis must be completed by someone with specific expertise 
and training) 

• Time consuming to score (each idea and relationship must be analyzed) 

• Participants must be taught how to construct a concept map 

• Can be difficult to determine commonalities (patterns) among participants or 
compare participant understandings 
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Document or Product Review 

What is it? 

Systematic examination of documents (e.g., rosters, time sheets, portfolios, photographs, 
participant work) collected during a project. 


How many respondents/participants? 

N/A 

Time issues? 

Varies - depends on the number of documents, their availability, and the amount of 
information being analyzed. 


Cost issues? 

Data collection relatively inexpensive, assuming that documents can be retrieved easily. 
Data analysis may be time consuming and therefore expensive. 


When to use it? 

Documents can be used to quantify particular project components (e.g., number of 
participants, amount of time spent on task, costs). Document review is unobtrusive and does 
not impact on the delivery of the project. Documents can provide multiple measures of the 
same construct, allowing evaluators to examine the process from more than one direction. 


What are some of the benefits? 

• Relatively cost effective 

• Assuming documents have been archived, data analysis can take place at any time 

• Documents can be used to monitor the development of a project 

• Project history can be documented 

• Data collection is often unobtrusive 

What are some of the limitations? 

• Collection and analysis of documents can be time consuming (especially if documents 
have not been archived in an orderly manner) 

• Available information may not match evaluation questions and needs 
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Case Study 

What is it? 

Holistic, detailed narrative describing a project or the experiences and perceptions of a 
particular participant. 


How many respondents/participants? 

One (in some cases 2-3) 


Time issues? 

Time intensive. Data collection, typically using ethnographic data collection methods, 
takes place consistently and continuously over an extended period of time. For the most 
part, data are qualitative and require a great deal of time to collect, organize and analyze. 
May require the evaluator to be a participant observer. 

Cost issues? 

Data collection and analysis requires a significant amount of time and expertise. 
Evaluator time could be costly. 

When to use it? 

Case studies are particularly useful when decision-makers or other stakeholders would 
like a detailed understanding of the project or the experience of a project participant, and 
are willing to forego generalizability. Case studies provide a rich, in-depth description of 
a participant or a small group of participants, or a project and how it works. Case studies 
provide vivid imagery, pulling a variety of data gathered from interviews, documents, 
and observation into a cohesive whole. Patterns of experience can be highlighted. Case 
studies can help decision-makers understand project successes and failures within a well- 
described context. 

What are some of the benefits? 

• Provides a holistic understanding of the project 

• Provides vivid imagery (narrative, paints a picture of the project) 

• Detailed understanding of patterns and experiences 


What are some of the limitations? 

• Time intensive 

• Detailed understanding of a particular participant, results cannot be generalized 

• Strategies for data collection and analysis are complex and require extensive training 
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Appendix B. 

Determining Sample Size 



The size of the sample is not nearly as important as the adequacy of the data collection 
design. Collecting large amounts of data from undefined sources is not recommended. 

If there is a bias in the data, it is unlikely to go away with more data collection. In 
conducting a survey, it is more important to obtain a representative sample than a large 
sample. Identify the groups to be sampled and put more effort into obtaining a high 
response rate (e.g., by phoning or sending reminders) rather than sending out large 
numbers of questionnaires and having a few undefined volunteers return them. 

If time and resources permit, a more rigorous sampling scheme should be used. Identify 
a resource person who can assist the planning team with determining statistics such 
as sampling error, level of confidence or risk, the number of variables to be examined, 
and the degree of variability in the attributes being measured. With pilot data in hand, 
a resource person can work out the total sample size needed for the kind of accuracy 
desired. 

If time or expertise is a constraint, use the following common sense guidelines: 

• The size of sample needed depends upon two things: 1) how accurate the summary 
data needs to be (for example, for no sampling error at all, the entire population 
would need to be measured), and 2) how variable the data are. If the evaluator started 
measuring a variable and found that every measure was the same, there would be little 
need to continue repeating the measurements to increase accuracy. On the other hand, 
the more the data vary, the more data must be collected to get a reasonably accurate 
measure of the mean. 

• Another consideration is acceptability to participants and audiences. For example, in 
collecting the views of staff, it might not be acceptable to take a sample, even though a 
sample would appear to be statistically adequate. Every staff member might need to be 
heard so that no one feels left out and there is no suspicion of bias. 

• Sampling may also be unacceptable if it causes more disturbance than would 
measuring everybody. For example, it may disrupt a lesson more to withdraw six 
students than to the test the entire class. 
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Determining Sample 

Size- 

Rules of Thumb 

Population 

Sample 

50 or less 

50 or less 

500 or less 

approx. 200 

1,000 or less 

approx. 275 

10,000+ 

approx. 350 

U.S. population 

2,000 to 4,000 

Source: Fitz-Gibbon & Morris. (1987). 


Minimizing Possible Errors in Random Sampling 

Type 

Cause 

Remedies 

Sampling error 

Using a sample, not the entire population 
to be studied. Error that can occur from 
sampling individuals that do not estimate 
the population. 

Larger samples reduce but do not eliminate 
sampling error. 

Sample bias 

Some of those selected to participate 
did not do so or provided incomplete 
information. 

Repeated attempts to reach non-respondents. 
Comparison of characteristics of non¬ 
respondents to determine if any 
systematic differences exist. 

Response bias 

Responses do not reflect “true” opinions 
or behaviors because questions were 
misunderstood or respondents chose not 
to tell the truth. 

Careful pre-testing of instruments to revise 
misunderstood, leading, or threatening 
questions. No remedy exists for deliberate 
equivocation in self-administered interviews, 
but it can be spotted by careful editing. In 
personal interviews, this bias can be reduced 
by a skilled interviewer. 
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Appendix C 

Logic Models and Performance Measures 




c 




Logic models provide an easy starting point for the selection of meaningful and realistic 
measures of performance. How the overall project "works" must be understood in order 
to identify what needs to be measured. Logic models show how the project operates. 

Performance measures (or indicators) relate to how well a project performs, particularly 
with regard to the delivery of services (outputs) and achievement of results (outcomes). 
Indicators are used to represent the targets specified at each level of the TOP model. They 
are measurable characteristics of how well the targets are achieved and how well the 
project is performing. 

Logic models "flesh out" projects and allow planners to select what should be measured 
directly from the model. Using the logic model allows project planners to: 

• Select meaningful measures of performance. 

• Select performance measures from all levels of inputs, outputs, and outcomes. 

• Recognize how individual projects can contribute to the larger scale (program) goals. 


Common Types of Performance Measures (Indicators) 

Outcomes: 

• SEE Conditions: indices of water quality, public satisfaction, economic status 

• Practices: adoption of practices, use of recommendations 

• KASA: test scores, self-assessments, scale ratings 

• Reactions: participant ratings of project activities, level of satisfaction with subject matter 

Outputs: 

• Participation: attendance, volunteer leadership in project activities, engagement 

• Activities: frequency, duration, type of delivery, and content of activities 

Inputs: 

• Resources: amount of staff time, expenditures, and materials relative to project activities. 
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Logic Models as a Tool for Program Development and Evaluation 


The logic model provides a visual representation of the program and its evaluation. The logic model 
illustrates the relationships among the various program components: initial situation (e.g., degraded coastal 
areas with declining numbers of species), identified priorities (e.g., restoring coastal areas, increasing species 
diversity); inputs (i.e., resources needed to accomplish a set of activities); outputs (i.e„ activities designed 
to accomplish the program goal, as well as the audiences that participate in those activities); and short-term 
(immediate), medium-term (2-3 years), and long-term (4-10 years) outcomes-impacts. The logic model can 
help guide program planning, implementation, and evaluation. It can serve as a tool for clarifying program 
elements, identifying evaluation questions and indicators, and conducting ongoing self-evaluation. 


Logic Model, Evaluation Questions and Indicators 


INPUTS 




Example 


ID 


Staff 

Money 

Time 

Materials 

Partners 

Research 


Facilitator for 
workshop 
Field equipment 
Workshop 
materials and 
supplies 


D 

D 


OUTPUTS 

Activities Participants 

D 

OUTCOMES-IMPACTS 

Short-term Medium-term Long-term 






Workshops 

Publications 

Services 

Events 

Products 


Teachers 

Youths 

Community 

members 

D 

Increased 
knowledge 
Increased levels 
of skills 


Increased 
knowledge and 
skills used in 
appropriate 
settings 


Goal is reached 
and sustained 






Workshops 


Community 


Community 


Ongoing 


Species 

focusing 


members living 

r\ 

members who 


participation 


diversity in 

on coastal 


in coastal areas 

I) 

participate in 


in restoration 


restored areas 

restoration 



y 

workshops 


activities by 


increased 





understand 


community 







basics of coastal 


members 





restoration 






Evaluation Questions: What do you want to know? 


Were the inputs 
sufficient and timely? 
Did they meet the 
program goals? 


Did all activities 
occur as intended? 
What was the quality 
of the intervention? 
Was the content 
appropriate? 


Did targeted 
community members 
participate? Who 
did not participate? 
Who else was 
researched? 


Did knowledge 
increase? Did 
understanding of 
coastal restoration 
techniques increase? 
What else happened? 


Are community 
members continuing 
to participate in 
restoration activities? 
Are they participating 
in other activities? 


To what extent has 
the biodiversity of the 
targeted coastal area 
been increased? In 
what other ways has 
ecosystem quality 
increased? 


Indicators: How will you know it? 


Number of staff; 

Number of workshops 

Number of 

Number, percent 

Number, percent 

Number of species 

Funds invested; 

scheduled; 

community members 

with increased 

using new knowledge 

recovered; 

Delivery timetable 

Publications printed; 
Number of events 

participating; 

Number of teachers 
attending workshops 

knowledge of 
coastal restoration; 
Additional outcomes: 

+, - 

and skills to monitor 
progress of restoration 
activities; Additional 
outcomes: +, - 

Other positive 
environmental 
benefits; Additional 
outcomes: +, - 


Source: Powell, E Jones, L . and Henert, E. (2002). Enhancing Program Performance with Logic Models. Retrieved December 2003, from the University of Wisconsin- 
Extension web site: http://www.uwex.edu/ces/lmcourse 
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Appendix D 

Levels of Evaluation 








As we know from the TOP model there are a number of levels on which to evaluate 
projects. Each level provides different information about the processes and outcomes 
of a project. The project team should select the level of evaluation based on the type of 
information needed to evaluate the project accurately. The levels are discussed in more 
detail below: 


Scaling the program evaluation staircase 


SEE Conditions: Have targeted social, economic, or environ¬ 
mental conditions changed as a result of targeted changes in 
behavior? How have the participants been affected? 


Practices: Did participants’ behavior change as a result of the 
knowledge, attitudes, skills, or aspirations learned during project? 


KASA: What did the participants learn? Did participants improve skill or 
abilities? 


Reactions: What were the participants’ responses to the project or activity? 
Were participants satisfied with the project activities? 


Participation: How many of the targeted audience attended the project? Were partici¬ 
pants engaged in the activities? 


Activities: Were activities implemented as planned? What promotional strategies worked or 
failed? 


J Resources: Were targeted resources expended on the program (staff, time, money)? 

Source: Bennett, C. and Rockwell, K. (2004). Targeting Outcomes of Programs (TOP): A hierarchy for targeting outcomes and evaluating 
their achievement. Retrieved January 2009, from University of Nebraska-Lincoln website: http://citnews.unl.edu/TOP/index.html 


Level 1: Resources 

Evaluation at the resources level generally includes monitoring activities to explain the 
scope of the project in terms of dollars expended and staff time used. The most common 
way of doing this is to keep careful records so reporting this information is accurate. 
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Level 2: Activities 

Monitoring project activities (outputs) requires monitoring the amount of services, 
products, or activities delivered during a specified period of time. Keeping track of 
the amount of activities conducted, products developed, or services rendered is good 
management. Common examples include the number of professional development 
trainings delivered, the number classroom visits conducted, and the number education 
kits given away. Recall that process or implementation evaluation assesses the quality 
and quantity of activities planned with what was actually done. 


Level 3: Participation 

Participation is considered the active involvement by the targeted participants in 
education projects. Although recording attendance is certainly one method of monitoring 
the amount of time spent in an activity or project it is does not convey how "active" the 
participants were. Studies suggest that higher levels of participation can lead to desired 
outcomes. The more engaged or active the participants are in the project, the more likely 
they will be to make needed changes. 

Keeping records of intensity, duration, and breath of participation are all part of 
monitoring at the participation level. Intensity is the amount of time participants spend 
in a project or activity over a given period of time. Intensity is measured in terms of 
hours per day, days per week, and weeks per year. Duration summarizes the history 
of attendance, whether the participant is new to the project or has been attending for 
some time. Breath refers to the variety of activities participants attend within and across 
projects. Participants that are active in multiple projects may be more apt to achieve 
desired project outcomes. 


Level 4: Reactions 

Reactions measure the audiences' immediate positive or negative response to the project 
or learning experience. This is the most common level of outcome evaluation. Often 
referred to as "smile sheets," these evaluations ask participants to rate their perceptions 
about the quality and impact of the specific project or activity. Smile sheets can range 
from a handful of questions regarding the project delivery, facility, and/or usefulness, to 
forms that ask participants to rate all aspects of the activity. Reaction evaluations are an 
important tool to measure participants' satisfaction. They are relatively easy to administer, 
tabulate, and summarize in a results report 

Example questions to assess reactions: 

Attending the workshop was a: 

Poor use of my time 1.2.3.4.5 Good use of my time 

Length of workshop in relationship to the materials presented was: 

Too long 1.2.3 (just right).4.5 Too short 
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Workshop facilities were: 

Inadequate 1.2.3 (adequate).4.5 Great 

What were the strengths and weaknesses of the workshop? 

Level 5. KASA 

Evaluation at the KASA level measures whether participating in the project increases the 
audience's knowledge, attitudes, skills, and aspirations toward the issues. 

A number of different instruments or tools can be designed to measure what project 
participants have learned. Before and after tests, simulations or demonstrations, or other 
in-class methods allow instructors or project designers to determine if the knowledge or 
skills identified in the objectives were learned. It is important to remember that regardless 
of the method used to determine project outcomes at the KASA level, the "test" must 
relate directly to the course objectives. 

KASA level instruments are customized for every instructional activity or project and must 
reflect the conditions of the specific job or real-world application of the learning. It is also 
important to remember that KASA evaluations measure the level of knowledge or skills of 
participants at the time the test is administered. Unless the instruments are administered 
as part of a longitudinal evaluation design, they do not measure long-term knowledge 
or skill retention, nor are they an indication of how these will be applied to a real-world 
situation. 

Example questions to assess KASA: 

What was the most important thing you learned by participating in the workshop? 

List three benefits of an estuary: 

1. 

2 . 

3. 

True or False: An estuary is the same as a bay. 


Level 6. Practice or Behavior Change 

Questions at the behavior or practices level measure whether the participant has been 
able to use the new knowledge learned and skills aquired during a specific education 
project. 

Determining behavior changes is more complex than assessing at the reaction and KASA 
levels in that it requires contacting participants after they have had time to apply the 
new knowledge and skills. As with other evaluation levels, many different instruments 
can be used to collect the data. Each instrument has different strengths and limitations. 
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Instruments include surveys, interviews, focus groups, observations, and written 
document review. Regardless of the instrument, the questions should target specific skill 
and knowledge areas and ask participants if and how they have applied what they learned 
during the education project to these situations. Questions should focus on relevance of 
the program, whether participants have used the materials provided during the learning 
experience, how new knowledge has been applied, and examples of ways new skills have 
been applied. 

Measuring the application of the new knowledge and skills learned is becoming more 
accepted as an aspect of evaluation. It is important to know not only that participants 
understood the material during the learning experience, but that they were then able 
to go back to their homes, communities, or jobs and apply it. This level of evaluation 
provides evidence of whether transfer of learning has occurred. It is much more powerful 
to justify a project by demonstrating that participants used the information than by 
reporting the number of participants that "liked" the project. Many decision makers are 
now demanding this level of evaluation to account for the resources spent educating the 
target audience. 

Example questions to assess practice or behavior change: 

Have you applied the skills you learned at the workshop to your current projects? 

Not at all 1.2.3.4.5 Extensively 

Have you implemented the action plan you developed at the workshop? 

Not at all 1.2.3.4.5 Extensively 

Please describe one way you have used the materials in the past 6 months: 

Describe one component of your action plan that you have implemented fully: 

Have there been barriers in applying the information learned during the workshop? 

□ No 

□ Yes 

If yes, please explain your answer: 


Level 7. SEE Conditions 

Evaluating the impacts of the project on social, economic, or environmental conditions 
means measuring the degree to which KASA and behavior changes have affected the 
environment or the audience's lives. 

There is constant pressure within agencies to demonstrate the efficiency and effectiveness 
of their projects. In order to actually conclude that a project has had its desired effect, 
the participants have to "successfully" apply the new skills or knowledge. That is, the 
application of new skills and knowledge leads to the desired result or impact on an 
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audience or the environment. This level of long-term feedback is becoming increasingly 
important, particularly when priorities are being set or when decisions to continue or 
discontinue the project are being made. 

Measuring SEE conditions is typically feasible only for large-scale projects designed to 
produce specific results for a specific audience. For example, if the goal was to measure 
the results of teaching participants how to be a facilitator, the evaluator would need to 
focus on the people who experienced facilitation conducted by the project participants. 
This requires that data collection be at least one step removed from the initial 
participants in the project. Because it can be quite difficult to isolate the effect of the 
project, this level of evaluation can be complex and is not very common. 

Example questions to assess practice or behavior change: 

After training on estuary restoration: 

How many acres of estuary have been successfully restored? 

After a train-the-teacher workshop: 

To what extent have you incorporated ocean literacy principles into your curriculum? 

Not at all 1.2.3.4.5 Extensively 

Have student tests scores increased as a result of incorporating ocean literacy 
principles into your curriculum? 

Not at all 1.2.3.4.5 Extensively 
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Glossary 

Assessmen involves gathering data (either formally or informally) to be used in forming 
judgments (adapted from Mehrens and Lehman, 1991). 

Baseline data are collected at the beginning of the project prior to any services or activities 
being conducted. 

Capabil (PPBES definition) is the ability to do something. It is a combination of 
activities, processes, and skills. 

Capacity (PPBES definition) is the amount of resources needed to undertake a project (input 
capacity) or the amount of products created as a result of the project (output capacity). 
Evaluation is the systematic collection of information about activities, characteristics, and 
outcomes of projects to make judgments about the project, improve effectiveness, and/or 
inform decisions about future programming (adapted from Patton, 2002). 

Goal of a project is the ultimate, "big picture" impact desired. Goals are often difficult, if 
not impossible, to quantify. 

Items are individual questions on an instrument. 

Needs Assessmen' is a systematic investigation of an audience(s) to identify aspects of 
individual knowledge, skill, interest, attitude and/or abilities relevant to a particular issue, 
organization goal, or objective. 

Objective of a project is a specific measurable outcome desired. 

Population is the entire collection of individuals about whom one is trying to make 
accurate statements. 

Program represents a coordinated and systematic effort to address a part of the agency's 
mission. 

Project is an effort or activity focused on specific issues and audiences. A set of projects, 
taken together, supports a program. 

Qualitative Data are descriptive rather than enumerative. They are usually provided in 
the form of words, such as descriptions of events, transcripts of interviews, and written 
documents. Qualitative data can be transformed into quantitative data through coding 
procedures. 

Quantitative Data are numeric data. Analysis of quantitative data involves looking at 
relationships between quantities. 

Reliability is the extent to which a data gathering instrument measures a variable 
consistently time after time. 

Response rates are the percentage of a selected sample from which data were collected 
(responses actually received). A further calculation can sometimes be made of the 
fractions of the population requested in the sample. 

Sample is a subset of the population from which information is collected. 

Survey Instruments are any consistent method or tool by which information is 
systematically gathered. 

Validity of an instrument is the extent to which it measures what it purports to measure. A 
test may be valid for one purpose, but not another. 
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